Code

LoRA: zero_pad speed improvements#770

Code

Merged

carmocca merged 25 commits intoLightning-AI:mainLightning-AI/litgpt:mainfrom Andrei-Aksionov:lora_zero_pad_speed_improvementsAndrei-Aksionov/lit-gpt:lora_zero_pad_speed_improvementsCopy head branch name to clipboardMay 6, 2024

+55-39

Commits

Commits on Sep 18, 2023

• Use only index_copy without any views or reshapes.

  Andrei-Aksionov

  committed
  3a6e637

• Don't do transpose required for the merge method in each zero_pad call.

  Andrei-Aksionov

  committed
  1d591f2

• self.lora_ind as a property

  Andrei-Aksionov

  committed
  3d068cb

Commits on Sep 19, 2023

• Updates for a case with an inference tensor

  Andrei-Aksionov

  committed
  89d8eaa

• Docstring for a lora_ind property.

  Andrei-Aksionov

  committed
  6785012

• Reassign self._lora_ind so it will be recreated outside inference mode

  Andrei-Aksionov

  committed
  8951570

Commits on Sep 20, 2023

• Merge branch 'main' into lora_zero_pad_speed_improvements

  Andrei-Aksionov

  committed
  38d9ef0

Commits on Sep 22, 2023

• Make lora_ind property a bit shorter.

  Andrei-Aksionov

  committed
  eb73d27

• Trim comments for lora_ind property.

  Andrei-Aksionov

  committed
  4cc6cb3

Commits on Oct 9, 2023

• Merge branch 'main' into lora_zero_pad_speed_improvements

  Andrei-Aksionov

  committed
  138acaa

• If validate is running in no_grad mode there is no need to clone ind

  Andrei-Aksionov

  committed
  7013fed

• Merge branch 'main' into lora_zero_pad_speed_improvements

  Andrei-Aksionov

  authored
  b494438

Commits on Oct 10, 2023

• Merge branch 'main' into lora_zero_pad_speed_improvements

  Andrei-Aksionov

  authored
  865f883

Commits on Oct 16, 2023

• Merge branch 'main' into lora_zero_pad_speed_improvements

  Andrei-Aksionov

  authored
  48a401b

Commits on Oct 25, 2023

• Merge branch 'main' into lora_zero_pad_speed_improvements

  Andrei-Aksionov

  authored
  5a14afc

Commits on Nov 13, 2023

• Merge branch 'main' into lora_zero_pad_speed_improvements

  Andrei-Aksionov

  committed
  c5f079f

Commits on Nov 23, 2023

• Revert "Minor tutorial updates"

  Show description for 4b20c88
  Andrei-Aksionov

  committed
  4b20c88

• Revert "Fix typo"

  Show description for daf0e4e
  Andrei-Aksionov

  committed
  daf0e4e

• Revert "Revert "Minor tutorial updates""

  Show description for 652d7bd
  Andrei-Aksionov

  committed
  652d7bd

• Merge branch 'main' into lora_zero_pad_speed_improvements

  Andrei-Aksionov

  committed
  b16e385

• Undo weirdly appeared typo.

  Andrei-Aksionov

  committed
  015f868

Commits on May 6, 2024

• Merge branch 'main' into lora_zero_pad_speed_improvements

  Andrei-Aksionov

  committed
  532a710

• Add FSDP test with empty_init=True

  Andrei-Aksionov

  committed
  1ef81f3

• Merge branch 'main' into lora_zero_pad_speed_improvements

  Andrei-Aksionov

  authored
  9560bc0

• Merge branch 'main' into lora_zero_pad_speed_improvements

  carmocca

  authored
  576de42