WASMFS and OPFS backend use case

[kevodwyer]

Emsdk 3.1.59

Use case:
I have a c++ program compile to WASM via emscripten that expects the filesystem to be setup before main() function is run. It is a requirement that any files generated by the program are available for the next time it is run.

Previously I was able to make good use of BrowserFS to persist changes. That project has now been deprecated (and in my case a regression between emscripten 2.0.31 and 2.032 caused different file behaviour that I was unable to isolate/solve).

I suspect that the flow I followed is similar to a lot of other projects:

The file system construction currently happens in a function referenced as Module.preRun: [initialSetup]
where:

    var initialSetup = function(){
        Module["addRunDependency"]("setup");

	// build and config BrowserFS backend objects
        //BrowserFS.initialize(mfs);
        //FS.mount(BFS, {root: '/root'}, '/root');
	//make API calls to populate the file system
	//ie FS.createDataFile(path, filename, contents, true, true);

	Module["removeRunDependency"]("setup");
    }

After which the main method is automatically called and the filesystem would be available to the program.
As an added bonus it was easy enough to use the FS api methods like FS.readFile(filePath, { encoding: 'binary' }) from javascript to extract files.

I am looking at some of the recent references to WASMFS and particularly OPFS.
Like: #18112 (comment)

Unfortunately I'm having difficulty translating the hints into the above flow.

• OPFS is not available at the preRun stage?
• how to call the FS api in the JS 'world' without blocking the main thread?

Would be grateful for an outline on how to achieve this. The more complete the example working code the better.

[sbc100]

@tlively

[tlively]

@kripken, do you remember the details here? IIRC, WasmFS's use of global constructors means some (or all?) functionality is not available before global constructors run, but I forget how that interacts with preRun.

The FS APIs are inherently synchronous, so in that sense they will always block the main thread when called from the main thread, but in many situations they run fast enough that it shouldn't be a problem. If you really want to avoid blocking the main thread at all, you can do your FS operations from a separate thread or worker.

[kripken]

About preRun, yes, this is different with the old FS. preRun executes before the wasm is even compiled, which is before there is compiled WasmFS code to use. So using WasmFS from preRun will not work in general.

We do "queue" a few operations from JS that are then executed later. That allows file preloading to be done in preRun. But we do not queue all operations (in general that would mean implementing a full FS in JS).

Users do have a hook to be able to run code during WasmFS startup, which can be used instead of preRun,

emscripten/system/include/emscripten/wasmfs.h

Lines 96 to 101 in e7e3b25

	// A hook users can do to run code during WasmFS startup. This hook happens
	// before file preloading, so user code could create backends and mount them,
	// which would then affect in which backend the preloaded files are loaded (the
	// preloaded files have paths, and so they are added to that path and whichever
	// backend is present there).
	void wasmfs_before_preload(void);