Code

Validation step3#21

Code

Open

kylematoba wants to merge 18 commits intomainswiss-ai/nanotron:mainfrom validation_step3swiss-ai/nanotron:validation_step3Copy head branch name to clipboard

+763-402

Commits

Commits on Nov 18, 2024

• add validation step, take out some steps that I don't think work correctly (like dtype != float, requires grad), make flash attention dep optional (still needs to be checked)

  kylematoba

  committed
  f587ee8

Commits on Nov 19, 2024

• fix FA, start to make validation step look like train step

  kylematoba

  committed
  d0facd6

• FA-less ROPE with FA signature

  kylematoba

  committed
  a821f1e

• Can do training without FA

  kylematoba

  committed
  d6d89ae

• work

  kylematoba

  committed
  ffb5fbf

Commits on Nov 20, 2024

• Fixed wrong lr initialization when loading checkpoints

  AleHD

  committed
  5c4c0c6

Commits on Nov 21, 2024

• Log against iteration_step

  kylematoba

  committed
  ace4d33

Commits on Nov 22, 2024

• Update src/nanotron/trainer.py

  Show description for 8659582
  

  kylematoba

  and

  AleHD

  authored
  8659582

• Update src/nanotron/trainer.py

  Show description for b8731e2
  

  kylematoba

  and

  AleHD

  authored
  b8731e2

Commits on Nov 25, 2024

• Merge branch 'validation_step3' of github.com:swiss-ai/nanotron into validation_step3

  kylematoba

  committed
  0fc0f4e

Commits on Nov 26, 2024

• Redo saving and loading

  kylematoba

  committed
  f73f33b

• Merge branch 'load_correct_lr' into validation_step3

  kylematoba

  committed
  c4effa9

• work

  kylematoba

  committed
  36d5819

Commits on Nov 28, 2024

• work

  kylematoba

  committed
  5329310

Commits on Nov 29, 2024

• tweaks

  kylematoba

  committed
  3ead1c7

• tweaks

  kylematoba

  committed
  2bc34de

Commits on Nov 30, 2024

• Merge branch 'validation_step3' of github.com:swiss-ai/nanotron into validation_step3

  kylematoba

  committed
  1182d75

Commits on Dec 1, 2024

• work

  kylematoba

  committed
  2e74edf