Changing the base Model

[zarick-create]

I changed the base model in Fluxgym to a variation of the existing one by editing the models.yaml file and coping my own model to my huggingface account so that it could download from there and use it for training.

Everything works fine technically, but with my new flux base model, it's much harder to train the model then with the standard one.
What could be the reasons behind this behaviour?