Skip to content
This repository has been archived by the owner on Jun 24, 2024. It is now read-only.

Support Separate Loading of Vocabulary or Tensors #445

Open
skirodev opened this issue Dec 13, 2023 · 0 comments
Open

Support Separate Loading of Vocabulary or Tensors #445

skirodev opened this issue Dec 13, 2023 · 0 comments

Comments

@skirodev
Copy link
Contributor

skirodev commented Dec 13, 2023

We can provide parameters during model loading, such as "vocab_only" and "tensor_only". Loading only the vocabulary can be useful in cases where only the tokenizer is needed, while loading only tensors would be suitable for certain visual models.

I'm open to hearing any suggestions or advice regarding whether it would be better to implement a separate TokenizerModel or make modifications within the existing Model. Any suggestions would be greatly appreciated.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant