Changes to enable CUDA graph for LLM

[vasunvidia]

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Collection: [Note which collection this PR will affect]

Changelog

• Add specific line by line info of high level changes in this PR.

Usage

• You can potentially add a usage example below

# Add a code snippet demonstrating how to use this 

Jenkins CI

To run Jenkins, a NeMo User with write access must comment jenkins on the PR.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

• New Feature
• Bugfix
• Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

• Related to # (issue)

[ericharper]

jenkins

[timmoon10]

This change is needed since NVIDIA/TransformerEngine#575 is changing te.Float8Tensor's API for transpose caching. See the discussion in NVIDIA/TransformerEngine#735 and NVIDIA/TransformerEngine#575.

This functionality may be restored in the future to enable a performance optimization with interleaved pipeline parallelism. In particular, the first microbatch is slower since it requires computing transposes, and the resulting load imbalances lead to GPU idling. Aligning the transposes across the pipeline parallel group helps ensure full utilization. See a similar optimization for distributed optimizer communication in NVIDIA/apex#1611 and NVIDIA/Megatron-LM@d2de701.

[timmoon10]

@vasunvidia observed no convergence when this logic was removed. I wonder if you are using the latest version of NVIDIA/TransformerEngine#575? The transpose logic is handled internally in TE, and if we do transposes here we'll do it twice.

I think we should figure out what's going on. If we wanted to update the transposes here, the logic would be:

        # Update transpose caches
        params = set(self.parameter(fragment) for fragment in fragments)
        for param in params:
            if is_float8tensor(param):
                param._reset_caches()
                param.transpose_2d(cache=True)

[ShriyaPalsamudram]

jenkins

[timmoon10]

LGTM, although you should sign the commits to pass DCO.

See #8918 (comment).

The base branch was changed.