Add mitch-ish 256 for LUMI #351
Conversation
| python scripts/train.py configs/v1_5-mix-medium-mitch-ish.yaml ${@} \ | ||
| --run_name=${SLURM_JOB_ID} \ | ||
| --global_train_batch_size=4096 \ | ||
| --max_duration=238418 |
There was a problem hiding this comment.
Yes, it is, for a batch size of 8M tokens.
|
|
||
| module load LUMI/22.08 partition/G | ||
|
|
||
| export OLMO_CONTAINER=llm-lumi_latest.sif |
There was a problem hiding this comment.
What's the name of that image?
There was a problem hiding this comment.
llm-lumi-torch32_latest.sif, I think.
| @@ -0,0 +1,59 @@ | |||
| #!/bin/bash | |||
| #SBATCH --job-name=v1.5-mix-medium-mitch-ish | |||
There was a problem hiding this comment.
Config is now called mitch7? Or mitch7-something_related_to_large_batch_size?
There was a problem hiding this comment.
I don't see any reason to rename the train config. But we could rename this script to v1_5-mix-medium-mitch-ish-large-batch-on-lumi.sh?
There was a problem hiding this comment.
v1_5-mix-medium-mitch-ish-large-batch-on-lumi is a bit unwieldy, no? When we talk about it, we just call it mitchish. All the configs are v1.5 now. 7 is shorter than medium.
There was a problem hiding this comment.
Renamed to mitch-ish-7b.sh.
|
Between this config and the on @ibeltagy wrote, which one are we keeping? |
We can delete the one on my branch, but give me a few minutes to compare both configs and leave comments for differences. |
|
The differences are:
|
It runs on MosaicML which is a better comparison to LUMI. The problem with Kempner is not enough nodes, so FSDP still takes a lot of memory, and so we have to do all of these other tricks like activation checkpointing. |
|
@ibeltagy good catch. That's actually been updated on the |
This adds a 256-node mitch-ish run for LUMI (2x the batch size). I think this will run as-is, but if not we'll have to try a different FSDP wrapping strategy.