Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Avoid calling _free_fp16_param_shard() too early #1164

Open
wants to merge 7 commits into
base: ngoyal_changes_for_pp_fp8
Choose a base branch
from
Open

Avoid calling _free_fp16_param_shard() too early #1164

wants to merge 7 commits into from

Commits on Jan 12, 2024

  1. Added reshard hook for frozen params in backward

    @awgu
    awgu committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    a4f02ef View commit details
    Browse the repository at this point in the history

Commits on Feb 21, 2024

  1. Avoid calling _free_fp16_param_shard() too early with PR 1159

    @jiecaoyu
    jiecaoyu committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    f2bb56f View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2024

  1. Added requires_grad check for params_with_grad method (#1171) 

    Co-authored-by: Jie Wang <[email protected]>
    @whbldhwj
    whbldhwj and whbldhwj authored Mar 25, 2024
    Configuration menu
    Copy the full SHA
    1307b1d View commit details
    Browse the repository at this point in the history

Commits on Apr 1, 2024

  1. Changed to only run reshard hook if all gradients computed (#1166) 

    * Changed to only run reshard hook if all gradients computed

* Fix decreasing it/s with multi-grad hook
    @awgu
    awgu authored Apr 1, 2024
    Configuration menu
    Copy the full SHA
    5faca97 View commit details
    Browse the repository at this point in the history

Commits on Apr 5, 2024

  1. Add cast input argument (#1175) 

    Co-authored-by: Jie Wang <[email protected]>
    @whbldhwj
    whbldhwj and whbldhwj authored Apr 5, 2024
    Configuration menu
    Copy the full SHA
    7bcbc80 View commit details
    Browse the repository at this point in the history

Commits on May 16, 2024

  1. Only perform cat() during last microbatch backward() within FlattenPa… 

    …ramsWrapper (#1184)
    @chrisxcai
    chrisxcai authored May 16, 2024
    Configuration menu
    Copy the full SHA
    9cbb4a7 View commit details
    Browse the repository at this point in the history

Commits on Jun 10, 2024

  1. [FSDPv1] Optimize memory usage for optimize_backward_concat=True (#1186) 

    * optimize memory

* clean up

* allocate GPU memory directly

* add comment

* rename fp32_grads to fp32_flat_grad

* update doc
    @chrisxcai
    chrisxcai authored Jun 10, 2024
    Configuration menu
    Copy the full SHA
    b73fffe View commit details
    Browse the repository at this point in the history