How to train the tinymodel(Like 300M or 150M)

[yongding-tao]

❓ The question

Hello,

Thank you for this project. I encountered an issue while trying to train a TinyModel for my research. The data paths in the config files under ./configs/tiny/ point to S3 locations, for example:
s3://ai2-llm/preprocessed/olmo-mix/v1_6-decontaminated/books/gpt-neox-olmo-dolma-v1_5/part-0-00000.npy.

I tried replacing s3:// with https:// based on the previous "Reproducibility" section mentioned in the README (which seems to have been removed), but this approach did not work.

Could you clarify the correct way to access these files or provide updated data paths or instructions? Furthermore, can you give a guidance on how to train a TinyModel using Olmo?

Thank you for your help!

[SimonSuster]

You should replace s3://ai2-llm/ with https://olmo-data.org/.

So, your example would be: https://olmo-data.org/preprocessed/olmo-mix/v1_6-decontaminated/books/gpt-neox-olmo-dolma-v1_5/part-0-00000.npy

[yongding-tao]

@SimonSuster Thank you for your advice. It actually helps me solve the problem.