Code

GPT3 MoE#4

Code

Open

haeggee wants to merge 55 commits intomainswiss-ai/nanotron-multilingual:mainfrom moeswiss-ai/nanotron-multilingual:moeCopy head branch name to clipboard

+3,536-139

Commits

Commits on Jun 26, 2024

• XGLM work in progress: Causal Attention and Positional Embeddings work

  AleHD

  committed
  7aa3940

• WIP: GPT arch almost done, hf->nt converters working perfectly for non-distributed inference

  AleHD

  committed
  78dd53c

Commits on Jul 9, 2024

• Added hf2nt frontend + tested training

  AleHD

  committed
  a74c71a

Commits on Jul 11, 2024

• Added nt2hf conversion + tests :)

  AleHD

  committed
  04eaef9

• precommit

  AleHD

  committed
  138da5f

Commits on Jul 15, 2024

• Merge pull request #1 from swiss-ai/gpt

  Show description for 35c43f7
  negar-foroutan

  authored
  35c43f7

Commits on Jul 16, 2024

• Added MultilingualNanoset Config

  TJ-Solergibert

  committed
  0485fd6

• Added MultilingualNanoset

  TJ-Solergibert

  committed
  539832a

• Added Language token

  TJ-Solergibert

  committed
  d9f0670

• Forgot the trainer ups

  TJ-Solergibert

  committed
  efe8720

• Fix minor errors. Everything works

  TJ-Solergibert

  committed
  25ad39b

• Updated config file with GPT2 tokenized datasets in RCP

  TJ-Solergibert

  committed
  d91f9e1

Commits on Jul 17, 2024

• Before lunch

  TJ-Solergibert

  committed
  d0c14e3

• After lunch

  TJ-Solergibert

  committed
  9cfc5ea

Commits on Jul 18, 2024

• Ready

  TJ-Solergibert

  committed
  eed7bce

• Merge pull request #2 from TJ-Solergibert/multilingual_nanoset

  Show description for da50231
  negar-foroutan

  authored
  da50231

Commits on Aug 2, 2024

• start documenting moe setup

  

  Alexander Hagele

  committed
  7a932f8

• base moe file

  haeggee

  committed
  f08a05e

Commits on Aug 5, 2024

• add todo

  haeggee

  committed
  fa06c0d

• gpt3_moe basis

  haeggee

  committed
  a9dba53

• add nn.linear to init for moe router

  haeggee

  committed
  2efffb8

• changes to pipeline for backward through aux losses

  haeggee

  committed
  57c58b7

• correct block costs and flops

  haeggee

  committed
  3967bee

• case of dict in pipelineblock

  haeggee

  committed
  bcb94cc

• option for GLU or normal MLP

  haeggee

  committed
  91acdc0

Commits on Aug 6, 2024

• init of linear layer in starcoder

  haeggee

  committed
  df3befc

Commits on Aug 8, 2024

• potential bug in pipeline block

  haeggee

  committed
  6edce83

Commits on Aug 15, 2024

• XGLM work in progress: Causal Attention and Positional Embeddings work

  

  AleHD

  authored and

  negar-foroutan

  committed
  7425167

• WIP: GPT arch almost done, hf->nt converters working perfectly for non-distributed inference

  

  AleHD

  authored and

  negar-foroutan

  committed
  42695ba

• Added hf2nt frontend + tested training

  

  AleHD

  authored and

  negar-foroutan

  committed
  6294aad

• Added nt2hf conversion + tests :)

  

  AleHD

  authored and

  negar-foroutan

  committed
  b469ee9

• precommit

  

  AleHD

  authored and

  negar-foroutan

  committed
  1b19ca2

• Added MultilingualNanoset Config

  

  TJ-Solergibert

  authored and

  negar-foroutan

  committed
  c1fabac

• Added MultilingualNanoset

  

  TJ-Solergibert

  authored and

  negar-foroutan

  committed
  086b50d

• Added Language token

  

  TJ-Solergibert

  authored and

  negar-foroutan

  committed
  1fe7445

• Forgot the trainer ups

  

  TJ-Solergibert

  authored and

  negar-foroutan

  committed
  fb6631a

• Fix minor errors. Everything works

  

  TJ-Solergibert

  authored and

  negar-foroutan

  committed
  a6eb1bd

• Updated config file with GPT2 tokenized datasets in RCP

  

  TJ-Solergibert

  authored and

  negar-foroutan

  committed
  ef3fac4

• Before lunch

  

  TJ-Solergibert

  authored and

  negar-foroutan

  committed
  49294f1

• After lunch

  

  TJ-Solergibert

  authored and

  negar-foroutan

  committed
  8a80e5a

• Ready

  

  TJ-Solergibert

  authored and

  negar-foroutan

  committed
  0fa1971

• Add multilingual validation (#3)

  Show description for 8b68126
  

  TJ-Solergibert

  authored and

  negar-foroutan

  committed
  8b68126

• correct logging of all losses

  haeggee

  committed
  d08c949

Commits on Aug 16, 2024

• minor bug fix when using bias

  haeggee

  committed
  d14315f

• bias init in case of use for moe

  haeggee

  committed
  5dc67fe

• sparse upcycling converter

  haeggee

  committed
  fad3497

• merge main into moe; also, adapt gpt3(-moe) for val logging

  haeggee

  committed
  100ebf4

• add example config

  haeggee

  committed
  f31a1a3

Commits on Sep 4, 2024

• small fixes

  haeggee

  committed
  a45dc35

Commits on Sep 6, 2024

• fix for eval

  haeggee

  committed
  bb768bb

• lighteval fix for multiling

  haeggee

  committed
  f8e30b4

• Update moe.md

  haeggee

  authored
  ac27ada

• Update moe.md

  haeggee

  authored
  328b8c2

Commits on Oct 19, 2024

• Moe converters (#5)

  Show description for 3bce1f4
  

  AleHD

  and

  negar-foroutan

  authored
  3bce1f4

Commits on Oct 23, 2024

• Merge remote-tracking branch 'origin/main' into moe

  haeggee

  committed
  0507c7f