[Question] Using transformers.js inside an Obsidian Plugin

[benjaminshafii]

I'm trying to run transfomer.js inside of Obsidian but running into some errors:

This code is triggering the issues:

class MyClassificationPipeline {
	static task = "text-classification";
	static model = "Xenova/distilbert-base-uncased-finetuned-sst-2-english";
	static instance = null;

	static async getInstance(progress_callback = null) {
		if (this.instance === null) {
			// Dynamically import the Transformers.js library
			console.log('before import')
			let { pipeline, env } = await import("@xenova/transformers");
			console.log('after import')

			// NOTE: Uncomment this to change the cache directory
			// env.cacheDir = './.cache';

			this.instance = pipeline(this.task, this.model, {
				progress_callback,
			});
		}

		return this.instance;
	}
}
export default MyClassificationPipeline;

// Comment out this line if you don't want to start loading the model as soon as the server starts.
// If commented out, the model will be loaded when the first request is received (i.e,. lazily).
// MyClassificationPipeline.getInstance();

Link to source

These are the lines that are calling the code above

Context about Obsidian plugins:

• Obsidian plugin is just a single imported js file.
• Most of the time it's bundled using esbuild.

In my case, this is my esbuild setup

---

How should I be tackling this, what would be the recommended way to bundle transformer.js?

[xenova]

Hi there 👋 - very cool idea! For some of the other build tools (like webpack), you can ignore certain modules (see here for an example). In particular, you'd need to ignore the onnxruntime-node and sharp modules.

Alternatively, if you can link to a CDN, you can try https://cdn.jsdelivr.net/npm/@xenova/[email protected]

[benjaminshafii]

cool i'll try those :)

[benjaminshafii]

Hi there 👋 - very cool idea! For some of the other build tools (like webpack), you can ignore certain modules (see here for an example). In particular, you'd need to ignore the onnxruntime-node and sharp modules.

Alternatively, if you can link to a CDN, you can try https://cdn.jsdelivr.net/npm/@xenova/[email protected]

This doesn't seem to work because Obsidian plugins are actually run inside of node environment

[xenova]

Oh actually I didn't even notice in the first image, it says it uses electron! I made an electron sample app a few months ago here. It should still work, but let me know if you run into any issues.

[benjaminshafii]

hi @xenova Thanks again for taking the time to answer.

I don't think I can use this tutorial.

Obsidian plugins are tricky because I don't control the electron bundling process.

the constraint that I'm operating under is: I need to provide a single javascript file (and nothing else).

this single js file is then interpreted by Obsidian which is an electron app.

when onnxruntime-node runs this line:

require(`../bin/napi-v3/${process.platform}/${process.arch}/onnxruntime_binding.node`);

in my case the pwd is:
/Applications/Obsidian.app/Contents/Resources/electron.asar/renderer

and this won't be able to resolve.

it's starting to look more and more like an issue with onnx, so I'll look into this. but if you have any idea i'd be happy to hear it

[logancyang]

Hi @ashgansh I'm also thinking about using it in an Obsidian plugin. Did you figure this out?

[Hugo-Persson]

Hello! Any progress on this? @ashgansh @xenova ?

[lhr0909]

I am trying to do the same, and I think the best way to go about it is to somehow load onnxruntime-web in the plugin and make use of WebAssembly. I am trying to see if I can force a browser build and make some edits in the library to read/write model files in the vault. Will report when I have made progress.

[benjaminshafii]

sorry guys haven't figured it out.

i decided to go for a server-client architecture, currently i allow my users to self-host the server easily (just two comands) and atm focused on OpenAI + Ollama

not sure when i will revisit transformer.js for my use case. would still be interested to make it scale my plugin horizontally.

[lhr0909]

I have found a way to load onnxruntime-web into the project. The trick is to avoid using node APIs and load WASM version of onnxruntime into the plugin. The best way I have found is to use esbuild-plugin-wat to load the WASM files as base64 strings and override the onnxruntime env to load the WASM as blobs instead of going for any network requests. I see the lib has node vs browser check and I have removed them in my local patch for simplicity.

Feel free to check this branch: https://github.com/lhr0909/obsidian-plugin-hypersearch/tree/wasm-loading

Now that the bundling process is stable, I am going to look into hacking a bit more into the downloading file process now.. Might need to download the model files into the vault separately to be safe. We will see.

[lhr0909]

@xenova Currently running into a very strange issue where it is saying that Worker is not a constructor once I have loaded in the sources. Other than it is not threaded, it is working. Would be cool if you could help or give some more instructions on this. Is there a build process I can use to include un-obfuscated ORT-web source code for better debugging?

If you install obsidian, you can import the produced plugin (main.js, manifest.json and styles.css) into the plugin folder of Obsidian.

https://docs.obsidian.md/Plugins/Getting+started/Build+a+plugin

Plugin in question:

https://github.com/lhr0909/obsidian-plugin-hypersearch/tree/webpack-onnx-worker-error

Also a thread in Obsidian forum: https://forum.obsidian.md/t/can-plugins-use-web-worker/81040/6