support GaLore

[batman-do]

Can repo support GaLore soon ?
https://github.com/jiaweizzhao/GaLore

[Andrei-Aksionov]

Hello @batman-do

It looks like GaLore is a drop-in replacement for the PyTorch optimizer, meaning that you can take any script that does pretraining/fine-tuning and replacement the part that defines optimizer.

[batman-do]

Hello @batman-do

It looks like GaLore is a drop-in replacement for the PyTorch optimizer, meaning that you can take any script that does pretraining/fine-tuning and replacement the part that defines optimizer.

Can u support fsdp + qlora soon , i stuck oom when max_seq_length large when fine-tune lora > 24gb with model 7b

[rasbt]

Can repo support GaLore soon ?

In the works via #1192 !