[Question] How to convert to ONNX a fine-tuned model

[mrddter]

Hi, we're playing with this library to see if it can be useful for our project. I find it very easy and well done (congratulations).

The idea is not to use it directly as a frontend library but via node.js.
We've tried scripting a model directly from HF (google/flan-t5-small) and it worked but we're having trouble using a fine-tuned model.

Here what we tried. We fine-tuned a model (again google/flan-t5-small) and then converted it using the onnx script (in README.md).

The script generated the following files:

onnx/decoder_model_quantized.onnx
onnx/decoder_model.onnx
onnx/encoder_model_quantized.onnx
onnx/encoder_model.onnx
config.json
generation_config.json
quantize_config.json
special_tokens_map.json
spice.model
tokenizer_config.json
tokenizer.json

But when we tried to use it it gave us this error:

local_files_only=true or env.allowRemoteModels=false and file was not found locally at ./models/google/flan-t5-small-2/onnx/decoder_model_merged_quantized.onnx

Some advice or useful doc/link?
Thanks

[xenova]

You should be able to use our already-converted version: Xenova/flan-t5-small. The error you are seeing is because the original repo doesn't have the merged version of the onnx file, which is an optimization that reduced model size by a factor of ~2x.

[mrddter]

thanks for the answer, I'll try starting from your model and let you know here!