Performance fine-tuning recipes for llama3 8b + 70b #11046

vysarge · 2024-10-25T22:36:41Z

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Collection: llm

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: Valerie Sarge <[email protected]>

Signed-off-by: vysarge <[email protected]>

nemo/collections/llm/recipes/llama3_70b.py

nemo/collections/llm/recipes/llama3_8b.py

Signed-off-by: Valerie Sarge <[email protected]>

nemo/collections/llm/gpt/model/llama.py

nemo/collections/llm/recipes/llama3_70b.py

cuichenx

Just a few small questions

nemo/collections/llm/recipes/llama3_70b.py

cuichenx · 2024-10-28T21:20:42Z

nemo/collections/llm/recipes/llama3_8b.py

@@ -273,7 +285,82 @@ def finetune_recipe(
        recipe.optim.config.lr = 5e-6
    elif peft_scheme.lower() == 'lora':
        recipe.peft = run.Config(LoRA)
+        recipe.peft.dim = 8


have you verified that dim=8 would not compromise accuracy?

Rank 8 follows the default from torchtune here; in tests with 8b and 70b loss looks to decrease smoothly across 50-100 train steps.

cuichenx · 2024-10-28T21:22:19Z

nemo/collections/llm/recipes/llama3_8b.py

+        Use this method with caution and only when you need maximum performance.
+        It may not be suitable for all hardware configurations or use cases.
+    """
+    recipe.trainer.strategy.tensor_model_parallel_size = 1


just checking, TP1 works for full finetuning as well?

With BF16 grad it should. Let me double-check.

Yep, both fit.

cuichenx · 2024-10-28T21:25:52Z

nemo/collections/llm/recipes/llama3_70b.py

+
+    # Sequence length settings in the model and dataset must agree
+    recipe.model.config.seq_length = seq_length
+    recipe.data.seq_length = seq_length


without packed sequence, we usually use seq_length=2048 (because most samples in Squad are well shorter than 2048 tokens)

Just altered this to use 4K for packed and 2K for unpacked by default.

Signed-off-by: Valerie Sarge <[email protected]>

cuichenx

LGTM

Signed-off-by: Valerie Sarge <[email protected]>

* llama3 finetuning perf recipes progress capture Signed-off-by: Valerie Sarge <[email protected]> * Small syntax fix Signed-off-by: Valerie Sarge <[email protected]> * syntax Signed-off-by: Valerie Sarge <[email protected]> * Apply isort and black reformatting Signed-off-by: vysarge <[email protected]> * Correct ddp setting Signed-off-by: Valerie Sarge <[email protected]> * Fix hasattr check Signed-off-by: Valerie Sarge <[email protected]> * bf16 grad Signed-off-by: Valerie Sarge <[email protected]> * Update configs for 8b + 70b Signed-off-by: Valerie Sarge <[email protected]> * Set wgrad_deferral_limit Signed-off-by: Valerie Sarge <[email protected]> --------- Signed-off-by: Valerie Sarge <[email protected]> Signed-off-by: vysarge <[email protected]> Co-authored-by: vysarge <[email protected]>

* llama3 finetuning perf recipes progress capture * Small syntax fix * syntax * Apply isort and black reformatting * Correct ddp setting * Fix hasattr check * bf16 grad * Update configs for 8b + 70b * Set wgrad_deferral_limit --------- Signed-off-by: Valerie Sarge <[email protected]> Signed-off-by: vysarge <[email protected]> Co-authored-by: Valerie Sarge <[email protected]> Co-authored-by: vysarge <[email protected]>

* llama3 finetuning perf recipes progress capture Signed-off-by: Valerie Sarge <[email protected]> * Small syntax fix Signed-off-by: Valerie Sarge <[email protected]> * syntax Signed-off-by: Valerie Sarge <[email protected]> * Apply isort and black reformatting Signed-off-by: vysarge <[email protected]> * Correct ddp setting Signed-off-by: Valerie Sarge <[email protected]> * Fix hasattr check Signed-off-by: Valerie Sarge <[email protected]> * bf16 grad Signed-off-by: Valerie Sarge <[email protected]> * Update configs for 8b + 70b Signed-off-by: Valerie Sarge <[email protected]> * Set wgrad_deferral_limit Signed-off-by: Valerie Sarge <[email protected]> --------- Signed-off-by: Valerie Sarge <[email protected]> Signed-off-by: vysarge <[email protected]> Co-authored-by: vysarge <[email protected]> Signed-off-by: Hainan Xu <[email protected]>

* llama3 finetuning perf recipes progress capture Signed-off-by: Valerie Sarge <[email protected]> * Small syntax fix Signed-off-by: Valerie Sarge <[email protected]> * syntax Signed-off-by: Valerie Sarge <[email protected]> * Apply isort and black reformatting Signed-off-by: vysarge <[email protected]> * Correct ddp setting Signed-off-by: Valerie Sarge <[email protected]> * Fix hasattr check Signed-off-by: Valerie Sarge <[email protected]> * bf16 grad Signed-off-by: Valerie Sarge <[email protected]> * Update configs for 8b + 70b Signed-off-by: Valerie Sarge <[email protected]> * Set wgrad_deferral_limit Signed-off-by: Valerie Sarge <[email protected]> --------- Signed-off-by: Valerie Sarge <[email protected]> Signed-off-by: vysarge <[email protected]> Co-authored-by: vysarge <[email protected]>

llama3 finetuning perf recipes progress capture

dbc6e6e

Signed-off-by: Valerie Sarge <[email protected]>

vysarge force-pushed the vsarge/ft_recipes branch from a38d021 to dbc6e6e Compare October 25, 2024 22:41

vysarge and others added 3 commits October 25, 2024 15:43

Small syntax fix

08146cf

Signed-off-by: Valerie Sarge <[email protected]>

syntax

b41c510

Signed-off-by: Valerie Sarge <[email protected]>

Apply isort and black reformatting

41bef5a

Signed-off-by: vysarge <[email protected]>

github-advanced-security bot found potential problems Oct 25, 2024

View reviewed changes

nemo/collections/llm/recipes/llama3_70b.py Fixed Show fixed Hide fixed

nemo/collections/llm/recipes/llama3_8b.py Fixed Show fixed Hide fixed

Correct ddp setting

054671a

Signed-off-by: Valerie Sarge <[email protected]>

erhoo82 added r2.0.0 Run CICD labels Oct 25, 2024

vysarge added 2 commits October 25, 2024 21:12

Fix hasattr check

26cd2db

Signed-off-by: Valerie Sarge <[email protected]>

bf16 grad

5906fc9

Signed-off-by: Valerie Sarge <[email protected]>

jiemingz reviewed Oct 28, 2024

View reviewed changes

nemo/collections/llm/gpt/model/llama.py Show resolved Hide resolved

Merge branch 'main' into vsarge/ft_recipes

c5b39e8

vysarge marked this pull request as ready for review October 28, 2024 19:15

JimmyZhang12 requested a review from cuichenx October 28, 2024 19:18

jiemingz approved these changes Oct 28, 2024

View reviewed changes

JimmyZhang12 reviewed Oct 28, 2024

View reviewed changes

nemo/collections/llm/recipes/llama3_70b.py Show resolved Hide resolved

cuichenx reviewed Oct 28, 2024

View reviewed changes

Update configs for 8b + 70b

6a7cd2a

Signed-off-by: Valerie Sarge <[email protected]>

vysarge requested a review from cuichenx October 28, 2024 22:18

cuichenx previously approved these changes Oct 28, 2024

View reviewed changes

Set wgrad_deferral_limit

445349f

Signed-off-by: Valerie Sarge <[email protected]>

vysarge dismissed cuichenx’s stale review via 445349f October 28, 2024 22:24

JimmyZhang12 approved these changes Oct 28, 2024

View reviewed changes

JimmyZhang12 added Run CICD and removed Run CICD labels Oct 28, 2024

JimmyZhang12 merged commit ce9f1dd into NVIDIA:main Oct 29, 2024
152 of 154 checks passed

ko3n1g mentioned this pull request Oct 29, 2024

Cherry pick Performance fine-tuning recipes for llama3 8b + 70b (11046) into r2.0.0 #11068

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Performance fine-tuning recipes for llama3 8b + 70b #11046

Performance fine-tuning recipes for llama3 8b + 70b #11046

vysarge commented Oct 25, 2024 •

edited

Loading

cuichenx left a comment

cuichenx Oct 28, 2024

vysarge Oct 28, 2024

cuichenx Oct 28, 2024

vysarge Oct 28, 2024

vysarge Oct 28, 2024

cuichenx Oct 28, 2024

vysarge Oct 28, 2024

cuichenx left a comment

Performance fine-tuning recipes for llama3 8b + 70b #11046

Performance fine-tuning recipes for llama3 8b + 70b #11046

Conversation

vysarge commented Oct 25, 2024 • edited Loading

What does this PR do ?

Changelog

Usage

GitHub Actions CI

Before your PR is "Ready for review"

Who can review?

Additional Information

cuichenx left a comment

Choose a reason for hiding this comment

cuichenx Oct 28, 2024

Choose a reason for hiding this comment

vysarge Oct 28, 2024

Choose a reason for hiding this comment

cuichenx Oct 28, 2024

Choose a reason for hiding this comment

vysarge Oct 28, 2024

Choose a reason for hiding this comment

vysarge Oct 28, 2024

Choose a reason for hiding this comment

cuichenx Oct 28, 2024

Choose a reason for hiding this comment

vysarge Oct 28, 2024

Choose a reason for hiding this comment

cuichenx left a comment

Choose a reason for hiding this comment

vysarge commented Oct 25, 2024 •

edited

Loading