How can I use Prodigy Optimiser?

[Reviem0]

I'd like to use Prodigyopt with the kohya Lora trainer colab, how can I add it in?

[Eden-Luminar]

Prodigy Optimiser is desingned to be used to train?

[Reviem0]

Thanks @MetroByte for the answer

If this is relevant for you: For "kohya-trainer" google colab:

1. Add a code block (on top, for example) with code !pip install -U prodigyopt
2. Proceed setup as usual up to 5.3
3. You have to change your settings a bit in 5.3-5.5:

• In optimizer_args set the following (you can change these if you know what are you doing): ["decouple=True", "weight_decay=0.01", "d_coef=2", "use_bias_correction=True", "safeguard_warmup=True"]
• unet_lr, text_encoder_lr to 1 (important)

5. Then you have to change your config files in folder LoRA/config. In config_file.toml:

• Make sure that every lr (learning rates) are set to (probably) 0.1 to 1, or not set at all (otherwise error, needs checking) Error when trying to train using DAdaptation optimizer bmaltais/kohya_ss#1160 (comment)
• optimizer_args is present
• optimizer_type is Prodigy (safecheck)

Then you have to change a bit in the kohya-trainer/train_network.py

1. At line ~12-13 add from prodigyopt import Prodigy
2. Find and comment line (with #) containing text optimizer_name, optimizer_args, optimizer = … (Line number ~224-227)
3. Add the following three lines below the commented line:

optimizer_name = "Prodigy" optimizer_args = "['decouple': True, 'weight_decay': 0.01, 'd_coef': 2, 'use_bias_correction': True, 'safeguard_warmup': True]" optimizer = Prodigy(trainable_params)

(first two are for metadata printing only though)

Then training should run fine (I hope)

If you re-run steps 5.2-5.4, config file will be replaced, so better to change config files directly.

Tested but I get overfitting results so early, I don't know how to set Prodigy correctly…