Add StragglerDetection and FTlauncher to NeMo2.0 #11117

ShriyaPalsamudram · 2024-10-31T14:44:32Z

What does this PR do ?

Add StragglerDetection and FTlauncher to NeMo2.0

Collection: [Note which collection this PR will affect]

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

nemo/collections/llm/recipes/callbacks/default.py

nemo/collections/llm/recipes/llama3_70b_64k.py

nemo/collections/llm/recipes/starcoder.py

nemo/collections/llm/recipes/callbacks/default.py

github-actions · 2024-11-01T21:22:05Z

[🤖]: Hi @ShriyaPalsamudram 👋,

We wanted to let you know that a CICD pipeline for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g

ericharper · 2024-11-04T20:09:45Z

@maanug-nv to introduce abstraction to reduce number of files changed to add a callback for all recipes.

github-actions · 2024-11-05T05:55:00Z

[🤖]: Hi @ShriyaPalsamudram 👋,

We wanted to let you know that a CICD pipeline for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g

nemo/lightning/run/plugins.py

tests/collections/llm/test_fault_nvrx.py

maanug-nv · 2024-11-14T23:55:08Z

nemo/lightning/run/plugins.py

+            assert any(
+                map(
+                    lambda callback: (
+                        callback.__fn_or_cls__ == PreemptionCallback if '__fn_or_cls__' in dir(callback) else False


Why does Nsys not work with Preemption? is this supposed to be FaultToleranceCallback ?

This assert is part of the ConfigValidationPlugin where we are validating Preemption callback. Only change here is to add if '__fn_or_cls__' in dir(callback) else False because there was some other callback that didn't have __fn_or_cls__ and hence this assert was failing.

Signed-off-by: ShriyaPalsamudram <[email protected]>

Signed-off-by: Shriya Palsamudram <[email protected]>

Signed-off-by: ShriyaPalsamudram <[email protected]>

Signed-off-by: Shriya Palsamudram <[email protected]>

Signed-off-by: ShriyaPalsamudram <[email protected]>

Signed-off-by: Shriya Palsamudram <[email protected]>

Signed-off-by: Shriya Rishab <[email protected]>

maanug-nv · 2024-11-19T17:38:46Z

tests/collections/llm/test_fault_nvrx.py

+        assert "Straggler report processing time" in log_content
+    if args.crash_step:
+        assert f"Exception: Simulating a crash at step {args.crash_step}!" in log_content
+


We should somehow confirm that we still finish the training despite the crash, right? Maybe the common.verify_ckpt_dir will be enough, it should ensure that there's a checkpoint for step 20.

github-actions · 2024-11-26T10:41:58Z

beep boop 🤖: 🚨 The following files must be fixed before merge!

Your code was analyzed with PyLint. The following annotations have been identified:


------------------------------------
Your code has been rated at 10.00/10

Thank you for improving NeMo's documentation!

github-actions · 2024-11-26T10:41:59Z

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

Your code was analyzed with PyLint. The following annotations have been identified:

************* Module nemo.lightning.run.plugins
nemo/lightning/run/plugins.py:81:0: C0301: Line too long (145/119) (line-too-long)
nemo/lightning/run/plugins.py:92:0: C0301: Line too long (201/119) (line-too-long)
nemo/lightning/run/plugins.py:102:0: C0301: Line too long (200/119) (line-too-long)
nemo/lightning/run/plugins.py:104:0: C0301: Line too long (127/119) (line-too-long)
nemo/lightning/run/plugins.py:188:0: C0301: Line too long (174/119) (line-too-long)
nemo/lightning/run/plugins.py:264:0: C0301: Line too long (150/119) (line-too-long)
nemo/lightning/run/plugins.py:71:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/run/plugins.py:112:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/run/plugins.py:166:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/run/plugins.py:211:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/run/plugins.py:274:4: C0116: Missing function or method docstring (missing-function-docstring)

-----------------------------------
Your code has been rated at 9.16/10

Thank you for improving NeMo's documentation!

ShriyaPalsamudram · 2024-12-02T08:58:31Z

Close PR as the tests are being added as part of #11406

ShriyaPalsamudram requested review from pablo-garay and ko3n1g as code owners October 31, 2024 14:44

ShriyaPalsamudram marked this pull request as draft October 31, 2024 14:44

github-advanced-security bot found potential problems Oct 31, 2024

View reviewed changes

nemo/collections/llm/recipes/callbacks/default.py Fixed Show fixed Hide fixed

ShriyaPalsamudram added Run CICD and removed Run CICD labels Oct 31, 2024

maanug-nv reviewed Nov 1, 2024

View reviewed changes

nemo/collections/llm/recipes/llama3_70b_64k.py Outdated Show resolved Hide resolved

maanug-nv reviewed Nov 1, 2024

View reviewed changes

nemo/collections/llm/recipes/starcoder.py Outdated Show resolved Hide resolved

maanug-nv reviewed Nov 1, 2024

View reviewed changes

nemo/collections/llm/recipes/callbacks/default.py Outdated Show resolved Hide resolved

ShriyaPalsamudram added Run CICD and removed Run CICD labels Nov 1, 2024

github-actions bot added the CI label Nov 1, 2024

ShriyaPalsamudram force-pushed the shriya/resiliency branch from 6382449 to a14a642 Compare November 4, 2024 19:53

ShriyaPalsamudram added Run CICD and removed Run CICD labels Nov 4, 2024

ShriyaPalsamudram force-pushed the shriya/resiliency branch from 598d275 to eeb9414 Compare November 4, 2024 22:08

ShriyaPalsamudram added Run CICD and removed Run CICD labels Nov 4, 2024

ericharper reviewed Nov 5, 2024

View reviewed changes

nemo/lightning/run/plugins.py Outdated Show resolved Hide resolved

ShriyaPalsamudram force-pushed the shriya/resiliency branch from 38fce0d to a92e667 Compare November 13, 2024 18:35

ShriyaPalsamudram added Run CICD and removed Run CICD labels Nov 13, 2024

github-advanced-security bot found potential problems Nov 13, 2024

View reviewed changes

tests/collections/llm/test_fault_nvrx.py Fixed Show fixed Hide fixed

tests/collections/llm/test_fault_nvrx.py Fixed Show fixed Hide fixed

tests/collections/llm/test_fault_nvrx.py Fixed Show fixed Hide fixed

tests/collections/llm/test_fault_nvrx.py Fixed Show fixed Hide fixed

maanug-nv reviewed Nov 14, 2024

View reviewed changes

ShriyaPalsamudram and others added 11 commits November 15, 2024 04:58

Apply isort and black reformatting

00aa2e1

Signed-off-by: ShriyaPalsamudram <[email protected]>

Revert enabling straggler_detection by default in all recipes

0aaa9e0

Signed-off-by: Shriya Palsamudram <[email protected]>

Remove unused imports

3fdcdce

Signed-off-by: Shriya Palsamudram <[email protected]>

Remove extra check in ConfigValidationPlugin

b26bd70

Signed-off-by: Shriya Palsamudram <[email protected]>

Address pylinter issues

98a533f

Signed-off-by: Shriya Palsamudram <[email protected]>

Improve straggler detection testing and add doc string

166048f

Signed-off-by: Shriya Palsamudram <[email protected]>

Apply isort and black reformatting

3d0a03d

Signed-off-by: ShriyaPalsamudram <[email protected]>

fix paths

bff35c4

Signed-off-by: Shriya Palsamudram <[email protected]>

Add assert for crash

e6bbb27

Signed-off-by: Shriya Palsamudram <[email protected]>

Apply isort and black reformatting

ba065bd

Signed-off-by: ShriyaPalsamudram <[email protected]>

Append run logs to a file after a crash

7f27da9

Signed-off-by: Shriya Palsamudram <[email protected]>

ShriyaPalsamudram force-pushed the shriya/resiliency branch from 442bd12 to 7f27da9 Compare November 15, 2024 12:59

ShriyaPalsamudram added the Run CICD label Nov 15, 2024

Set FAULT_TOL_FINISHED_FLAG_FILE and FAULT_TOL_CFG_PATH

6c0857f

Signed-off-by: Shriya Palsamudram <[email protected]>

ShriyaPalsamudram added Run CICD and removed Run CICD labels Nov 18, 2024

Merge branch 'main' into shriya/resiliency

fd55915

Signed-off-by: Shriya Rishab <[email protected]>

ShriyaPalsamudram added Run CICD and removed Run CICD labels Nov 19, 2024

ShriyaPalsamudram marked this pull request as ready for review November 19, 2024 16:20

maanug-nv reviewed Nov 19, 2024

View reviewed changes

maanug-nv force-pushed the shriya/resiliency branch from 1f08f43 to fd55915 Compare November 26, 2024 10:41

maanug-nv mentioned this pull request Nov 26, 2024

Add tests for resiliency feature integration #11406

Open

8 tasks

ShriyaPalsamudram closed this Dec 2, 2024

ShriyaPalsamudram deleted the shriya/resiliency branch December 2, 2024 08:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add StragglerDetection and FTlauncher to NeMo2.0 #11117

Add StragglerDetection and FTlauncher to NeMo2.0 #11117

ShriyaPalsamudram commented Oct 31, 2024 •

edited

Loading

github-actions bot commented Nov 1, 2024

ericharper commented Nov 4, 2024

github-actions bot commented Nov 5, 2024

maanug-nv Nov 14, 2024

ShriyaPalsamudram Nov 15, 2024

maanug-nv Nov 19, 2024

github-actions bot commented Nov 26, 2024

github-actions bot commented Nov 26, 2024

ShriyaPalsamudram commented Dec 2, 2024

Add StragglerDetection and FTlauncher to NeMo2.0 #11117

Add StragglerDetection and FTlauncher to NeMo2.0 #11117

Conversation

ShriyaPalsamudram commented Oct 31, 2024 • edited Loading

What does this PR do ?

Changelog

Usage

GitHub Actions CI

Before your PR is "Ready for review"

Who can review?

Additional Information

github-actions bot commented Nov 1, 2024

ericharper commented Nov 4, 2024

github-actions bot commented Nov 5, 2024

maanug-nv Nov 14, 2024

Choose a reason for hiding this comment

ShriyaPalsamudram Nov 15, 2024

Choose a reason for hiding this comment

maanug-nv Nov 19, 2024

Choose a reason for hiding this comment

github-actions bot commented Nov 26, 2024

github-actions bot commented Nov 26, 2024

ShriyaPalsamudram commented Dec 2, 2024

ShriyaPalsamudram commented Oct 31, 2024 •

edited

Loading