Question about the OLMo2 Stage 2 training procedures: was the optimizer state from Stage 1 used during the training of Stage 2?

[Taoer1996]

❓ The question

Thank you for the fully open-sourced OLMo2; that's really amazing research work!!

And we have a small question regarding the training process of Stage 2. Was the optimizer state from Stage 1 used during the training of Stage 2? If so, will the full checkpoint, including the optimizer state, be open-sourced?

[aman-17]

Hey @Taoer1996, we loaded the final checkpoint from Stage-1 for Stage-2, and the optimizer state wasn’t reset. As for open-sourcing the full checkpoints with the optimizer state, we are planning to release them, I’ll keep you posted if there are any updates on this.

[Taoer1996]

Hey @Taoer1996, we loaded the final checkpoint from Stage-1 for Stage-2, and the optimizer state wasn’t reset. As for open-sourcing the full checkpoints with the optimizer state, we are planning to release them, I’ll keep you posted if there are any updates on this.

Thanks for clarifying that! Really excited to see what comes next.