tokenizer.model

[hasakikiki]

I fine-tuned an llm based on the llama skeleton and used convert_hf_checkpoint and quantize to complete the quantification. However, when generating, the tokenizer.model file is missing. How can I operate or generate it?

[yanboliang]

Which model are you going to inference? If it's a Llama 3+ model, we already copied it to the right place in convert_hf_checkpoint.

gpt-fast/scripts/convert_hf_checkpoint.py

Lines 118 to 126 in c9f683e

	if 'llama-3' in model_name.lower():
	if 'llama-3.1-405b' in model_name.lower():
	original_dir = checkpoint_dir / "original" / "mp16"
	else:
	original_dir = checkpoint_dir / "original"
	tokenizer_model = original_dir / "tokenizer.model"
	tokenizer_model_tiktoken = checkpoint_dir / "tokenizer.model"
	print(f"Copying {tokenizer_model} to {tokenizer_model_tiktoken}")
	shutil.copy(tokenizer_model, tokenizer_model_tiktoken)