Request: Fine-Tuning Support for Any GGUF Model

[Spider-netizen]

Hi @ggerganov,

Is there any chance to add support for fine-tuning for any GGUF model? I’m especially interested in fine-tuning gemma2.

Thank you for your work!

[dspasyuk]

@Spider-netizen why not just use a base model in Llama factory and then convert it to gguf? With Lora, you can finetune a 7B model with as low as 16 Gb of Vram.

[Spider-netizen]

My hardware is very limited at the moment, but I'll look into it, thanks.

[ggerganov]

Hopefully we will support this in the future, but it will take time to get there (currently we don't even have enough resources to maintain the existing training/finetuning examples)

[Dampfinchen]

I think BitNet will allow for really resource efficient fine tuning, and most importantly won't degrade quality as much as training it in 4 bit.

[Spider-netizen]

To my understanding BitNet models only use the same architicture of the model but they need to be trained from scratch basically. I only want to slightly fine tune an existing model to make it better.

[gtygo]

In the case of local computation (limited computing resources), I think RAG combined with gguf can produce good results.

[ngxson]

The finetune example is currently not very usable, so we temporary remove it ==> #8669

For now, you can do QLoRA finetune via huggingface PEFT and bitsandbytes. IMO it costs a bit of time to setup, but easier to use.

The output adapter can be converted to gguf via convert_lora_to_gguf.py and can be loaded without merging to model.

Support for switching different lora will be added to server soon.

[Spider-netizen]

Thank you for the detailed response and suggestions. Unfortunately, it seems bitsandbytes doesn't support my current hardware.