Is this LoRA implementation correct?

[AayushManShrestha]

I was trying to fine-tune Moondream2 by using LoRA. But, I got weird loss curves.
Here is the link to the code: LoRA-finetune
Here is the loss curve when doing fine-tuning normally:

Here is the loss curve when using LoRA for fine-tuning