How to pretrain Llama-2 with tokenmonster

[jerryjalapeno]

Hello, how would we go about pretraining a llama2 model from scratch? All the existing codebases and documentation use a BPE based model, so how would we use tokenmonster instead? Is there a drop in replacement, or do we need to edit the training codebases on our own? Specifically, I am using axolotl to pretrain.

[Sovenok-Hacker]

To do it you need VERY VERY MUCH computation power, are you sure?
But if you have it and really want to do it, it can be done by creating a custom tokenizer for HF Transformers. I didn`t researched it deeply, but Transformers support .vocab format, it will be easy to use TokenMonster in HF Transformers, and then connect it to Axolotl.