Remove generate/lora

[awaelchli]

The special script generate/lora.py or litgpt generate lora is no longer needed as of #1081, because automatic merging takes place during finetuning in the final checkpoint. The docs were already updated.

We could in theory keep this, but the benefits are very minor.

Pros for keeping lora.py:

• Educational purposes
• Slightly faster because you don't need to merge + save checkpoint first (but on the other hand merging is a one-time thing).

Cons for keeping lora.py:

• We don't have a chat/lora.py anyway, would we add it?
• More duplicated code to maintain
• The generate script has still all lora hyperparameters exposed, which is error prone. We would have to fix this like we did for merging to use the correct parameters automatically (extra work but not a real con)

Alternative:

• If it's confusing to have generate {base, adapter} but no lora, we could also keep the lora CLI command but reroute it to base.py

[rasbt]

One thing is that the lora.py generate script show the elegance of LoRA where you never have to modify the original pretraining model. That being said, in practice, I don't think it is that much of an advantage in terms of runtime or memory resources, because you have to always run the pretrained model (merged or unmerged, it's the same size) anyways.

The only thing is that merging could maybe require additional memory but I don't think so because of the lazy merging.

Overall, I am in strong favor of removing it, because it will simplify the code base and leave us with less code to maintain like you said. It will also be easier for writing tutorials etc. Let's remove it :).

[lantiga]

Let's remove for now @awaelchli, we can revisit in the future if we want to serve multiple lora layers or increase velocity for users, but that would require a proper flow