QLoRA subcommand #1119
QLoRA subcommand #1119
Conversation
Co-authored-by: rasbt <[email protected]> Co-authored-by: Carlos Mocholí <[email protected]>
Co-authored-by: Sebastian Raschka <[email protected]>
Co-authored-by: Carlos Mocholí <[email protected]>
Co-authored-by: awaelchli <[email protected]> Co-authored-by: Carlos Mocholí <[email protected]>
Co-authored-by: awaelchli <[email protected]>
Co-authored-by: awaelchli <[email protected]>
Co-authored-by: awaelchli <[email protected]>
Co-authored-by: awaelchli <[email protected]>
Co-authored-by: awaelchli <[email protected]> Co-authored-by: Carlos Mocholí <[email protected]>
litgpt/__main__.py
Outdated
| "finetune.qlora.lora_query": True, | ||
| "finetune.qlora.lora_key": True, | ||
| "finetune.qlora.lora_value": True, | ||
| "finetune.qlora.lora_projection": True, | ||
| "finetune.qlora.lora_mlp": True, | ||
| "finetune.qlora.lora_head": True, |
There was a problem hiding this comment.
Since this is not what we have in the configs, I'd like to first rerun the finetuning and update the configs together here.
Co-authored-by: awaelchli <[email protected]> Co-authored-by: Carlos Mocholí <[email protected]>
|
@rasbt Do you know if this subcommand is still necessary? Or should I close this? |
|
I think that if we go the |
|
Code is magic and anything is possible, but in general, you are correct. Note also that to support that kind of notation, a replacement for jsonargparse will be necessary because its argument introspection capabilities are not complex enough to support that notation. I tried in the past unsuccessfully. |
qlora = lora with bnb and true precision.
I think bf16-true is fine as a default because bitsandbytes doesnt' support the old cards well (like my laptop)