Controlnet training explain

[hinablue]

Could you please explain how the controlnet training really do ?

[laksjdjf]

There is a simple link available for Colab, but it lacks information about ControlNet.

The required directory structure is as follows:

dataset_directory
  |----buckets.json # bucketing.py
  |----latents
  |  |----hoge.npy # latent.py
  |----captions
  |  |----hoge.caption # tagger.py if anime style
  |----control 
  |  |----hoge.png # no code for it

1. Adjust the resolution of images using bucketing.py

python3 preprocess/bucketing.py \
--input_dir "image_directory" \
--output_dir "dataset_directory" \
--resolution 768 \
--min_length 512 \
--max_length 1024 \
--max_ratio 2 \
--threads 2

2. Cache latent

python3 preprocess/latent.py \
--directory "dataset_directory" \
--output_path "dataset_directory/latents" \
--model "diffusers_path" \
--batch_size 4

4. Put caption in captions dir
   The caption file with the same filename as the image and with the extension ".caption"

5. Put controlnet guide images in control dir
   Same filename and resolution as the image preprocessed by bucketing.py.

6. Edit setting file
   The example config file for training ControlNet can be found at config/controlnet.yaml

7. Train
   python3 main.py config/controlnet.yaml and model is saved in trained/controlnet/

(Optional) convert controlnet model for webui
python3 tools/convert_controlnet.py trained_path output_file_name