New File API

We want to design a new File API which is importable as ESM module for example:

import { File } from 'cockpit/file';

This is the new way of doing things similar to fsinfo:

import { fsinfo } from "cockpit/fsinfo";

This new API would directly support two new features which we need in Cockpit and Files:

setting the owner/group of a (newly created) file
setting the mode of a (newly created) file

The reason we can't fit this into the old API easily is that cockpit.file('/tmp/foo').replace("text", undefined, "root", "root") is already allowed in the existing API. JavaScript has no issues allowing extra unused arguments, so these extra arguments won't be passed through the bridge and even if they did extra unknown arguments don't trigger any ProtocolError.

Another reason to start with a new API is to remove the use of our jQuery styled promises which aren't compatible with the native Promise API.

Compatibility with the old fsreplace1 API

As we want to support new features in our new API we have a descision to make:

Don't support the new API with an old bridge, introducing fsreplace2
Best effort support, try to support new features but raise Errors when we can't. Our projects then handle this in fallback code
Offer a capibility option, so a developer knows when they can switch to a new API
Offer fallback code in the new API, means we keep some legacy around. We already have such code atm for setting owner/group in cockpit-files
Offer a shim? Like sdl-compat, you use the new API internally but with the old API

The issue boils down to our pages bundling cockpit.js which supports newer features then the current bridge might support. We need a way to detect this either:

Let a channel advertise capabilities?
We introduce a new channel?

What is needed from the File API:

creating files with contents, ownership, permissions
modifying file contents, ownership, permissions
reading file contents
being able to take an existing tag when modifying/creating
deleting an file - now supported by sending a magic zero byte.
stat -> fsinfo
symlink? :)
rename? - currently being handled in files with mv
watching for updates (Although fsinfo is used when available in the new code)

Do we still support?

syntax_object
Watching

cockpit.file() most used API

.read()
.replace()

cockpit.spawn() usages which do file I/O

const conf_path = "/etc/systemd/timesyncd.conf.d/50-cockpit.conf";
const custom_ntp_config_file = cockpit.file(conf_path, { superuser: "require" });
// this must be readable with tight umask, timesyncd runs as unprivileged user
await cockpit.spawn(["mkdir", "-p", "-m755", "/etc/systemd/timesyncd.conf.d"], { superuser: "require" });
await custom_ntp_config_file.replace(text);
await cockpit.spawn(["chmod", "644", conf_path], { superuser: "require" });

cockpit.spawn(["mkdir", "-p", memoryLocation], { superuser, err: "message" })

        const opts = { err: "message", superuser: "require" } as const;
        await cockpit.spawn(["mkdir", path], opts);
        await cockpit.spawn(["chown", owner, path], opts);
    } else {
        await cockpit.spawn(["mkdir", path], { err: "message" });

        try {
            await cockpit.spawn(["mkdir", "-p", config_dir]);
        } catch (err) {
            const exc = err as cockpit.BasicError; // HACK: You can't easily type an error in typescript
            addAlert(_("Unable to create bookmark directory"), AlertVariant.danger, "bookmark-error",
                     exc.message);
            return;
        }

await cockpit.file(destination, { superuser: "try" }).replace("");
await cockpit.spawn(["chown", owner, destination], { superuser: "try" });
await cockpit.file(destination, { superuser: "try" }).read()
    .then((((_content: string, tag: string) => {
	options = { superuser: "try", tag };
    }) as any /* eslint-disable-line @typescript-eslint/no-explicit-any */));
const stat = await cockpit.spawn(["stat", "--format", "%a", destination], { superuser: "try" });

try {
    await cockpit.spawn(["chown", owner, filename], { superuser: "require" });
} catch (err) {
    console.warn("Cannot chown new file", err);
    try {
	await cockpit.spawn(["rm", filename], { superuser: "require" });
    } catch (err) {
	console.warn(`Failed to cleanup ${filename}`, err);
    }
    throw err;
}

Design

class File(filename: str, args?: { superuser, binary: true, max_read_size?: 8 * 1024 }) {
  remove(tag?: str): Promise<void>;
  replace(content: str, { owner?: str | int, group?: str | int, mode?: int, tag?: str}): Promise<tag: str, mode: int, owner/group>
  read(): Promise<{content: str, tag: str}>;
  watch(callback({content: str, tag: str}), {read: bool}): function();
}

const {content, tag} = await File('/tmp/foo').read();

modify() -> either return the tag or make users use fsinfo to query this information
watch() returns a handle so we can remove() the watch. What to do in new API
do we keep close() to close pending reads() like for example read /dev/random or /dev/zero

As cockpit.file().read() is not streamed

Meeting notes

Checking if new options are supported in fsreplace1 is impossible so we need to change the bridge -> Easiest solution is to introduce fsreplace2 and reject any unknown option! Cockpit.js would try fsreplace2 and fallback to fsreplace1.
The new ESM module should work for simple cases but fall with owner/mode options and basically still use the existing code in files.
We don't feel like keeping syntax_object supported

Plan

Implement new class
Native promises
QUnit tests

implement remove
implement read

File streaming with `fsread1`

Because this becomes a stream, we can't return the tag realistically so this becomes a totally different API apart from streaming.

async function* asyncEvent(object, event) {
  let defer
  const next = (event) => defer.resolve(event)
  object.on(event, next)
  try {
    while (true) {
      defer = Promise.withResolvers()
      yield await defer.promise
    }
  } finally {
    object.off(event, next)
  }
}

async * read(options?: ReadOptions): Promise<ReadData> {
	let channel;
	const opts = { ...this.get_options(), payload: 'fsread1' };
	const data: StrOrBytes[] = [];
	let binary = false;
	console.log(opts);

	if (opts.binary) {
	    binary = true;
	    channel = new ReadChannel<Uint8Array>({ ...opts, binary: true });
	} else {
	    channel = new ReadChannel(opts);
	}

	let defer;
	const next = (event) => defer.resolve(event);
	channel.on('data', next);
	channel.on('close', message => defer.reject(message));

	// ready
	await channel.wait();

	try {
	    while (true) {
		defer = Promise.withResolvers();
		yield await defer.promise;
	    }
	} catch (exc) {
	    console.log(exc);
	}
}

const large_file = new File(large_filename);
for await (const data of large_file.read()) {
	console.log(data);
}

Questions

how to use the new API in tests! We want a qunit development workflow
generic way to convert eventlistener to async?
we require listening to ready message for fsreplace1 now
do we want reads to be abortable? (So an abortcontroller support)
what about AsyncIterator, could implement streaming reads, now we batch the data in the client. Downside a bit harder to read, needs prototyping
stream API? seems we need something to replace potential cockpit.file().addEventListener() users, check if there are any
discuss error handling in promises, do we stick with BasicError?
how do we handle the existing API for "Atomic modifications" => that's cockpit.file().modify()
Channel.wait_close() or extend wait() API
Make it a static Channel class? Remove global options

Notes