support new TOKENIZER-TYPE=EXTERNAL_EMBEDDINGS_FROM_DICT#387

Merged

bensha6757 merged 4 commits intomasterfrom side/bens/embeddings-injection-tokenizer

Dec 15, 2024

+57-23

Commits on Dec 10, 2024

support new TOKENIZER-TYPE=EXTERNAL_EMBEDDINGS_FROM_DICT
Ben Shapira
committed
add default sub tokenizer and improve logic
Ben Shapira
committed
simplified ifs
Ben Shapira
committed

Commits on Dec 15, 2024

adding comments for the default sub-tokenizer
Ben Shapira
committed