Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[PT FE] Improve 16bit patching #27693

Open
wants to merge 2 commits into
base: releases/2024/5
Choose a base branch
from
Open

[PT FE] Improve 16bit patching #27693

wants to merge 2 commits into from

Commits on Nov 22, 2024

  1. [PT FE] Inherit signature from forward while patching (openvinotoolki… 

    …t#27413)

### Details:
 - *Inherit signature from forward while patching*

### Tickets:
 - *ticket-id*

---------

Signed-off-by: Maxim Vafin <[email protected]>
    @mvafin
    mvafin committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    d8484e8 View commit details
    Browse the repository at this point in the history

  2. [PT FE] Optimize memory usage of patch_model (openvinotoolkit#27428) 

    ### Details:
- *`no_jit_trace` was using extra memory to get and store trace state,
which contained all graph that was produced before, that increase memory
consumption of tracing. For example for FLUX model it used about 20Gb of
extra memory*
- *Saving `args` on meta device didn't work without `no_jit_trace`. To
workaround this issue we now pass args directly to forward without
saving them in `Trampoline`. This allows better flow for arguments and
reduce memory used to save those args. However this changes the behavior
of `evaluate` of `ModuleExtension`, because now it uses the args that
were passed to `convert` and not the original args.*

optimum-cli for FLUX with `torch_dtype=torch.bfloat16` before change: 

![image](https://github.com/user-attachments/assets/f070068a-e52e-4558-956e-95afa64d1dbc)
optimum-cli for FLUX with `torch_dtype=torch.bfloat16` after change:

![image](https://github.com/user-attachments/assets/a76fe1df-2410-4b92-9b01-38ef40133b2b)

Note: optimum doesn't yet support `torch_dtype=torch.bfloat16` for FLUX.

### Tickets:
 - *CVS-151254*

---------

Signed-off-by: Maxim Vafin <[email protected]>
    @mvafin
    mvafin committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    0f93623 View commit details
    Browse the repository at this point in the history