Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Gemma2 in Nemo2 with Recipes #11037

Merged
merged 16 commits into from
Oct 30, 2024
Merged

Gemma2 in Nemo2 with Recipes #11037

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
8250e02
add gemma2 in nemo2.0 and 2b recipe
suiyoubi Oct 25, 2024
931cb09
Apply isort and black reformatting
suiyoubi Oct 25, 2024
3683ff8
Fix gemma1 inference bug
suiyoubi Oct 25, 2024
6ee8b0f
add more recipe
suiyoubi Oct 25, 2024
225ff9c
minor fix
suiyoubi Oct 25, 2024
d6e8fe9
recipe fix
suiyoubi Oct 25, 2024
ebd79a7
Apply isort and black reformatting
suiyoubi Oct 25, 2024
0b0736b
merge fix
suiyoubi Oct 25, 2024
a34cb88
Merge branch 'main' into aot/gemma2
suiyoubi Oct 27, 2024
eb4fa31
Merge branch 'main' into aot/gemma2
suiyoubi Oct 27, 2024
b47e4fe
Merge branch 'main' into aot/gemma2
suiyoubi Oct 28, 2024
efd426f
Merge branch 'main' into aot/gemma2
cuichenx Oct 29, 2024
cf33e03
Merge branch 'main' into aot/gemma2
suiyoubi Oct 29, 2024
213ffca
move import to avoid te issue
suiyoubi Oct 29, 2024
3b58df6
Apply isort and black reformatting
suiyoubi Oct 29, 2024
cf4f3a9
Merge branch 'main' into aot/gemma2
suiyoubi Oct 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions nemo/collections/llm/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,11 @@
CodeLlamaConfig13B,
CodeLlamaConfig34B,
CodeLlamaConfig70B,
Gemma2Config,
Gemma2Config2B,
Gemma2Config9B,
Gemma2Config27B,
Gemma2Model,
GemmaConfig,
GemmaConfig2B,
GemmaConfig7B,
Expand Down Expand Up @@ -165,6 +170,11 @@
"CodeGemmaConfig2B",
"CodeGemmaConfig7B",
"GemmaModel",
"Gemma2Model",
"Gemma2Config9B",
"Gemma2Config",
"Gemma2Config27B",
"Gemma2Config2B",
"Baichuan2Config",
"Baichuan2Config7B",
"Baichuan2Model",
Expand Down
12 changes: 12 additions & 0 deletions nemo/collections/llm/gpt/model/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,13 @@
GemmaConfig7B,
GemmaModel,
)
from nemo.collections.llm.gpt.model.gemma2 import (
Gemma2Config,
Gemma2Config2B,
Gemma2Config9B,
Gemma2Config27B,
Gemma2Model,
)
from nemo.collections.llm.gpt.model.hf_auto_model_for_causal_lm import HfAutoModelForCausalLM
from nemo.collections.llm.gpt.model.llama import (
CodeLlamaConfig7B,
Expand Down Expand Up @@ -142,6 +149,11 @@
"CodeGemmaConfig2B",
"CodeGemmaConfig7B",
"GemmaModel",
"Gemma2Config",
"Gemma2Config27B",
"Gemma2Config2B",
"Gemma2Config9B",
"Gemma2Model",
"LlamaModel",
"Baichuan2Config",
"Baichuan2Config7B",
Expand Down
2 changes: 1 addition & 1 deletion nemo/collections/llm/gpt/model/gemma.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ def make_vocab_size_divisible_by(vocab_size):
rotary_base=source.rope_theta,
gated_linear_unit=True,
make_vocab_size_divisible_by=make_vocab_size_divisible_by(source.vocab_size),
share_embeddings_and_output_weights=False,
share_embeddings_and_output_weights=True,
fp16=(dtype_from_hf(source) == torch.float16),
bf16=(dtype_from_hf(source) == torch.bfloat16),
params_dtype=dtype_from_hf(source),
Expand Down
Loading
Loading