From 94ef2fcd68ba73444fabb323bb9f0544c4de75cf Mon Sep 17 00:00:00 2001 From: Dirk Groeneveld Date: Tue, 26 Nov 2024 17:57:59 -0800 Subject: [PATCH 1/7] More checkpoint information --- README.md | 46 +++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 41 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index ae6ad095d..cf33eeed2 100644 --- a/README.md +++ b/README.md @@ -39,12 +39,48 @@ pip install ai2-olmo ### Overview -The core models in the OLMo family released are: -| Model | Training Tokens | Context Length | Training Config | W&B Logs | -|-------|-----------------|:--------------:|-----------------|----------| -| [OLMo2 7B](https://huggingface.co/allenai/OLMo-2-1124-7B) | 4 Trillion | 4096 | [configs/official-1124/OLMo2-7B-stage1.yaml](https://github.com/allenai/OLMo/blob/main/configs/official-1124/OLMo2-7B-stage1.yaml) | wandb.ai/…/OLMo2-7B (link to come) -| [OLMo2 13B](https://huggingface.co/allenai/OLMo-2-1124-13B) | 5 Trillion | 4096 | [configs/official-1124/OLMo2-13B-stage1.yaml](https://github.com/allenai/OLMo/blob/main/configs/official-1124/OLMo2-13B-stage1.yaml) | wandb.ai/…/OLMo2-13B (link to come) +OLMo pretraining follows a two-stage training procedure. +In the first stage, we train on large amounts of mostly web-based data: [OLMo-mix-1124](https://huggingface.co/datasets/allenai/olmo-mix-1124) +In the second stage, we train on a smaller amount of high-quality, targeted data: [Dolmino-mix-1124](https://huggingface.co/datasets/allenai/dolmino-mix-1124) +#### Stage 1 + +To get the tokenized training data, look at the paths in the training configs. +To reproduce at large scale, we recommend downloading the files locally and changing the paths to point to your +local file system, for performance reasons. + +| | OLMo2 7B | OLMo2 13B | +|-----------------|-------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------| +| Number of tokens| 4 Trillion | 5 Trillion | +| Checkpoint | [stage1-step928646-tokens3896B](https://huggingface.co/allenai/OLMo-2-1124-7B/tree/stage1-step928646-tokens3896B) | [stage1-step596057-tokens5000B](https://huggingface.co/allenai/OLMo-2-1124-13B/tree/stage1-stage1-step596057-tokens5000B) | +| Training config | [OLMo2-7B-stage1.yaml](https://github.com/allenai/OLMo/blob/main/configs/official-1124/OLMo2-7B-stage1.yaml) | [configs/official-1124/OLMo2-13B-stage1.yaml](https://github.com/allenai/OLMo/blob/main/configs/official-1124/OLMo2-13B-stage1.yaml) | +| WandB | wandb.ai/…/OLMo2-7B (link to come) | wandb.ai/…/OLMo2-13B (link to come) | + +#### Stage 2 for the 7B + +For the 7B model, we train three times with different data order on 50B high quality tokens, and then average ("soup") the models. + +| | Checkpoint | Training config | WandB | +|----------------------|-------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|--------------| +| random seed 42 | [stage2-ingredient1-step11931-tokens50B](https://huggingface.co/allenai/OLMo-2-1124-7B/tree/stage2-ingredient1-step11931-tokens50B) | | link to come | +| random seed 42069 | [stage2-ingredient2-step11931-tokens50B](https://huggingface.co/allenai/OLMo-2-1124-7B/tree/stage2-ingredient2-step11931-tokens50B) | | link to come | +| random seed 666 | [stage2-ingredient3-step11931-tokens50B](https://huggingface.co/allenai/OLMo-2-1124-7B/tree/stage2-ingredient3-step11931-tokens50B) | | link to come | +| *final souped model* | [main](https://huggingface.co/allenai/OLMo-2-1124-7B/tree/main) | | link to come | + +#### Stage 2 for the 13B + +For the 13B model, we train three times with different data order on 100B high quality tokens, and one more time +on 300B high quality tokens. Then we average ("soup") the models. + +| | Checkpoint | Training config | WandB | +|------------------------|----------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|--------------| +| random seed 1110, 100B | [stage2-ingredient1-step11931-tokens100B](https://huggingface.co/allenai/OLMo-2-1124-13B/tree/stage2-ingredient1-step11931-tokens100B) | | link to come | +| random seed 2662, 100B | [stage2-ingredient2-step11931-tokens100B](https://huggingface.co/allenai/OLMo-2-1124-13B/tree/stage2-ingredient2-step11931-tokens100B) | | link to come | +| random seed 6209, 100B | [stage2-ingredient3-step11931-tokens100B](https://huggingface.co/allenai/OLMo-2-1124-13B/tree/stage2-ingredient3-step11931-tokens100B) | | link to come | +| random seed 2662, 300B | [stage2-ingredient4-step11931-tokens300B](https://huggingface.co/allenai/OLMo-2-1124-13B/tree/stage2-ingredient4-step11931-tokens300B) | | link to come | +| *final souped model* | [main](https://huggingface.co/allenai/OLMo-2-1124-13B/tree/main) | | link to come | + +#### Instruction tuned variants For instruction tuned variants of these models, go to * [OLMo2 7B Instruct](https://huggingface.co/allenai/OLMo-2-1124-7B-Instruct) From fbc7e02e29903bac3d75f2c33bd49109ec5bbafc Mon Sep 17 00:00:00 2001 From: Dirk Groeneveld Date: Tue, 26 Nov 2024 17:59:03 -0800 Subject: [PATCH 2/7] Let's be bold --- README.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index cf33eeed2..2a9fd173c 100644 --- a/README.md +++ b/README.md @@ -60,12 +60,12 @@ local file system, for performance reasons. For the 7B model, we train three times with different data order on 50B high quality tokens, and then average ("soup") the models. -| | Checkpoint | Training config | WandB | -|----------------------|-------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|--------------| -| random seed 42 | [stage2-ingredient1-step11931-tokens50B](https://huggingface.co/allenai/OLMo-2-1124-7B/tree/stage2-ingredient1-step11931-tokens50B) | | link to come | -| random seed 42069 | [stage2-ingredient2-step11931-tokens50B](https://huggingface.co/allenai/OLMo-2-1124-7B/tree/stage2-ingredient2-step11931-tokens50B) | | link to come | -| random seed 666 | [stage2-ingredient3-step11931-tokens50B](https://huggingface.co/allenai/OLMo-2-1124-7B/tree/stage2-ingredient3-step11931-tokens50B) | | link to come | -| *final souped model* | [main](https://huggingface.co/allenai/OLMo-2-1124-7B/tree/main) | | link to come | +| | Checkpoint | Training config | WandB | +|------------------------|-------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|--------------| +| random seed 42 | [stage2-ingredient1-step11931-tokens50B](https://huggingface.co/allenai/OLMo-2-1124-7B/tree/stage2-ingredient1-step11931-tokens50B) | | link to come | +| random seed 42069 | [stage2-ingredient2-step11931-tokens50B](https://huggingface.co/allenai/OLMo-2-1124-7B/tree/stage2-ingredient2-step11931-tokens50B) | | link to come | +| random seed 666 | [stage2-ingredient3-step11931-tokens50B](https://huggingface.co/allenai/OLMo-2-1124-7B/tree/stage2-ingredient3-step11931-tokens50B) | | link to come | +| **final souped model** | [main](https://huggingface.co/allenai/OLMo-2-1124-7B/tree/main) | | link to come | #### Stage 2 for the 13B @@ -78,7 +78,7 @@ on 300B high quality tokens. Then we average ("soup") the models. | random seed 2662, 100B | [stage2-ingredient2-step11931-tokens100B](https://huggingface.co/allenai/OLMo-2-1124-13B/tree/stage2-ingredient2-step11931-tokens100B) | | link to come | | random seed 6209, 100B | [stage2-ingredient3-step11931-tokens100B](https://huggingface.co/allenai/OLMo-2-1124-13B/tree/stage2-ingredient3-step11931-tokens100B) | | link to come | | random seed 2662, 300B | [stage2-ingredient4-step11931-tokens300B](https://huggingface.co/allenai/OLMo-2-1124-13B/tree/stage2-ingredient4-step11931-tokens300B) | | link to come | -| *final souped model* | [main](https://huggingface.co/allenai/OLMo-2-1124-13B/tree/main) | | link to come | +| **final souped model** | [main](https://huggingface.co/allenai/OLMo-2-1124-13B/tree/main) | | link to come | #### Instruction tuned variants From b3b308838f2e579458fb39080ce28f10c0a2cb70 Mon Sep 17 00:00:00 2001 From: Dirk Groeneveld Date: Tue, 26 Nov 2024 18:00:38 -0800 Subject: [PATCH 3/7] Make links consistent --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 2a9fd173c..a641987e5 100644 --- a/README.md +++ b/README.md @@ -53,7 +53,7 @@ local file system, for performance reasons. |-----------------|-------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------| | Number of tokens| 4 Trillion | 5 Trillion | | Checkpoint | [stage1-step928646-tokens3896B](https://huggingface.co/allenai/OLMo-2-1124-7B/tree/stage1-step928646-tokens3896B) | [stage1-step596057-tokens5000B](https://huggingface.co/allenai/OLMo-2-1124-13B/tree/stage1-stage1-step596057-tokens5000B) | -| Training config | [OLMo2-7B-stage1.yaml](https://github.com/allenai/OLMo/blob/main/configs/official-1124/OLMo2-7B-stage1.yaml) | [configs/official-1124/OLMo2-13B-stage1.yaml](https://github.com/allenai/OLMo/blob/main/configs/official-1124/OLMo2-13B-stage1.yaml) | +| Training config | [OLMo2-7B-stage1.yaml](https://github.com/allenai/OLMo/blob/main/configs/official-1124/OLMo2-7B-stage1.yaml) | [OLMo2-13B-stage1.yaml](https://github.com/allenai/OLMo/blob/main/configs/official-1124/OLMo2-13B-stage1.yaml) | | WandB | wandb.ai/…/OLMo2-7B (link to come) | wandb.ai/…/OLMo2-13B (link to come) | #### Stage 2 for the 7B From 52f9985e8f180b98f7bbe8bab4ef60c06a0248f2 Mon Sep 17 00:00:00 2001 From: Dirk Groeneveld Date: Tue, 26 Nov 2024 18:01:24 -0800 Subject: [PATCH 4/7] Fix link --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index a641987e5..baf618763 100644 --- a/README.md +++ b/README.md @@ -52,7 +52,7 @@ local file system, for performance reasons. | | OLMo2 7B | OLMo2 13B | |-----------------|-------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------| | Number of tokens| 4 Trillion | 5 Trillion | -| Checkpoint | [stage1-step928646-tokens3896B](https://huggingface.co/allenai/OLMo-2-1124-7B/tree/stage1-step928646-tokens3896B) | [stage1-step596057-tokens5000B](https://huggingface.co/allenai/OLMo-2-1124-13B/tree/stage1-stage1-step596057-tokens5000B) | +| Checkpoint | [stage1-step928646-tokens3896B](https://huggingface.co/allenai/OLMo-2-1124-7B/tree/stage1-step928646-tokens3896B) | [stage1-step596057-tokens5000B](https://huggingface.co/allenai/OLMo-2-1124-13B/tree/stage1-step596057-tokens5000B) | | Training config | [OLMo2-7B-stage1.yaml](https://github.com/allenai/OLMo/blob/main/configs/official-1124/OLMo2-7B-stage1.yaml) | [OLMo2-13B-stage1.yaml](https://github.com/allenai/OLMo/blob/main/configs/official-1124/OLMo2-13B-stage1.yaml) | | WandB | wandb.ai/…/OLMo2-7B (link to come) | wandb.ai/…/OLMo2-13B (link to come) | From cf1ebbddf715294b7c3b4bb682efed0e9e20bb28 Mon Sep 17 00:00:00 2001 From: Dirk Groeneveld Date: Tue, 26 Nov 2024 18:02:24 -0800 Subject: [PATCH 5/7] Fix link again --- README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index baf618763..4a2690f61 100644 --- a/README.md +++ b/README.md @@ -49,12 +49,12 @@ To get the tokenized training data, look at the paths in the training configs. To reproduce at large scale, we recommend downloading the files locally and changing the paths to point to your local file system, for performance reasons. -| | OLMo2 7B | OLMo2 13B | -|-----------------|-------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------| -| Number of tokens| 4 Trillion | 5 Trillion | -| Checkpoint | [stage1-step928646-tokens3896B](https://huggingface.co/allenai/OLMo-2-1124-7B/tree/stage1-step928646-tokens3896B) | [stage1-step596057-tokens5000B](https://huggingface.co/allenai/OLMo-2-1124-13B/tree/stage1-step596057-tokens5000B) | -| Training config | [OLMo2-7B-stage1.yaml](https://github.com/allenai/OLMo/blob/main/configs/official-1124/OLMo2-7B-stage1.yaml) | [OLMo2-13B-stage1.yaml](https://github.com/allenai/OLMo/blob/main/configs/official-1124/OLMo2-13B-stage1.yaml) | -| WandB | wandb.ai/…/OLMo2-7B (link to come) | wandb.ai/…/OLMo2-13B (link to come) | +| | OLMo2 7B | OLMo2 13B | +|-----------------|-------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------| +| Number of tokens| 4 Trillion | 5 Trillion | +| Checkpoint | [stage1-step928646-tokens3896B](https://huggingface.co/allenai/OLMo-2-1124-7B/tree/stage1-step928646-tokens3896B) | [stage1-step596057-tokens5001B](https://huggingface.co/allenai/OLMo-2-1124-13B/tree/stage1-step596057-tokens5001B) | +| Training config | [OLMo2-7B-stage1.yaml](https://github.com/allenai/OLMo/blob/main/configs/official-1124/OLMo2-7B-stage1.yaml) | [OLMo2-13B-stage1.yaml](https://github.com/allenai/OLMo/blob/main/configs/official-1124/OLMo2-13B-stage1.yaml) | +| WandB | wandb.ai/…/OLMo2-7B (link to come) | wandb.ai/…/OLMo2-13B (link to come) | #### Stage 2 for the 7B From 2d17285ebd8b7e779c1f10576c0d09c39cd75e1d Mon Sep 17 00:00:00 2001 From: Dirk Groeneveld Date: Tue, 26 Nov 2024 18:03:29 -0800 Subject: [PATCH 6/7] Let's try relative links --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 4a2690f61..ca908a1a1 100644 --- a/README.md +++ b/README.md @@ -53,7 +53,7 @@ local file system, for performance reasons. |-----------------|-------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------| | Number of tokens| 4 Trillion | 5 Trillion | | Checkpoint | [stage1-step928646-tokens3896B](https://huggingface.co/allenai/OLMo-2-1124-7B/tree/stage1-step928646-tokens3896B) | [stage1-step596057-tokens5001B](https://huggingface.co/allenai/OLMo-2-1124-13B/tree/stage1-step596057-tokens5001B) | -| Training config | [OLMo2-7B-stage1.yaml](https://github.com/allenai/OLMo/blob/main/configs/official-1124/OLMo2-7B-stage1.yaml) | [OLMo2-13B-stage1.yaml](https://github.com/allenai/OLMo/blob/main/configs/official-1124/OLMo2-13B-stage1.yaml) | +| Training config | [OLMo2-7B-stage1.yaml](configs/official-1124/OLMo2-7B-stage1.yaml) | [OLMo2-13B-stage1.yaml](configs/official-1124/OLMo2-13B-stage1.yaml) | | WandB | wandb.ai/…/OLMo2-7B (link to come) | wandb.ai/…/OLMo2-13B (link to come) | #### Stage 2 for the 7B From 9ee1b6c8c19c657157ab32e93576b7b5d81d9ac1 Mon Sep 17 00:00:00 2001 From: Dirk Groeneveld Date: Tue, 26 Nov 2024 18:05:24 -0800 Subject: [PATCH 7/7] Fix link --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ca908a1a1..77b087c7b 100644 --- a/README.md +++ b/README.md @@ -77,7 +77,7 @@ on 300B high quality tokens. Then we average ("soup") the models. | random seed 1110, 100B | [stage2-ingredient1-step11931-tokens100B](https://huggingface.co/allenai/OLMo-2-1124-13B/tree/stage2-ingredient1-step11931-tokens100B) | | link to come | | random seed 2662, 100B | [stage2-ingredient2-step11931-tokens100B](https://huggingface.co/allenai/OLMo-2-1124-13B/tree/stage2-ingredient2-step11931-tokens100B) | | link to come | | random seed 6209, 100B | [stage2-ingredient3-step11931-tokens100B](https://huggingface.co/allenai/OLMo-2-1124-13B/tree/stage2-ingredient3-step11931-tokens100B) | | link to come | -| random seed 2662, 300B | [stage2-ingredient4-step11931-tokens300B](https://huggingface.co/allenai/OLMo-2-1124-13B/tree/stage2-ingredient4-step11931-tokens300B) | | link to come | +| random seed 2662, 300B | [stage2-ingredient4-step11931-tokens300B](https://huggingface.co/allenai/OLMo-2-1124-13B/tree/stage2-ingredient4-step35773-tokens300B) | | link to come | | **final souped model** | [main](https://huggingface.co/allenai/OLMo-2-1124-13B/tree/main) | | link to come | #### Instruction tuned variants