Diffusers Backend - ControlNet, Second Pass, Adetailer & Upscaling

[gohan2091]

With the recent update, I am trying to move away from the original backend and use the diffusers backend so I can use 1.5 and XL checkpoints together (and the slight speed boost is nice). Moving to the diffusers backend has disabled several extensions including ControlNet. I am aware there is now a Control tab which looks similar to ControlNet, however I am struggling to use it like I did with the standard ControlNet extension. For example....

My initial generative workflow once I found a prompt I want to experiment with is as follows (inside txt2Img):

1. Select a decent 1.5 checkpoint, select the CFG, steps, sampler etc. Set the dimensions to 512x768 (or 768x512)
2. Select Second Pass (hires.fix), select a latent upscaler (usually nearest exact), make the upscale to: x2 (so image is 1536/1024 / 1024x1536)
3. Enable Adetailer to automatic inpainting of face on each generation

Then I would run how many batches I want until I find images I really like.

I can do all this fine with the diffusers backend BUT sometimes I would have a Controlnet set as well while generating images. Maybe openpose, sometimes depth, it all depends.

Question 1) How do I do this in the diffusers backend? How do I have ControlNet function running while I am generating images with Second Pass (latent upscaler x2) plus Adetailer? I don't see this as an option in the Control menu. Am I missing something?

Question 2) Since Ultimate SD Upscaler script and Multidiffusion are both not working with diffusers backend, how do I go about upscaling my images? I've always had access to img2img upscaling on it's own but I used these two methods above (together with ControlNet Tiles) because I consider them better options. I've tried the Control menu and selected under ControlNet "tile" but for the processor I've no idea what to select. I was expecting to see tile_resample. I've left it as "none". Is there a guide on how to use the Control tab? This: https://github.com/vladmandic/automatic/wiki/Control#controlnet is not really helping me :(

Question 3) Where are the DPM++ 3M samplers? DPM++ 2M SDE?. I can still use these if I load an image and transfer the settings but they are missing from the dropdown menu.

I understand with my questions, I can go back to the original backend but then I don't have the ability to use XL models or other things. I would appreciate help with this from someone kind enough to offer a couple of minutes of their time. Or perhaps I should post on the Discord group?

[vladmandic]

Question 1) How do I do this in the diffusers backend? How do I have ControlNet function running while I am generating images with Second Pass (latent upscaler x2) plus Adetailer? I don't see this as an option in the Control menu. Am I missing something?

right now, you cant. control module is very new and doesn't yet have all options needed, specifically to run with adetailer.
so its either adetailer via standard text/image or control for controlnet.

Question 2)
i never used controlnet tile.

Question 3) Where are the DPM++ 3M samplers? DPM++ 2M SDE?. I can still use these if I load an image and transfer the settings but they are missing from the dropdown menu.

this has been answered several times, search.
samplers have been broken down into sampler + settings for sampler as having a flat list got us to 50+ samplers which is unmanageable.
so you select sampler and then set settings for it - for example, SDE in DPM* is a solver type, not a separate sampler.
and no, you cant use those if you load an image - they are what you DESIRE, doesn't mean that they're actually going to be used - and you can see that in generated image metadata.

[gohan2091]

Adetailer is just one thing but the ability to use ControlNet in a latent second pass is very important to me so it looks like I cannot use Diffusers backend because of this. Please consider if you can adding this functionality back.

If someone here can help me with question 2, I would appreciate.

Regarding the samplers, I did already search before I asked and it's really not very clear to me. Even what you wrote now doesn't help. In the UI, how do I set the solver type? I see mentions of solver type in the sampler settings in DPM solver algorithm such as sde-dpmsolver++ but it's not clear to me what I select. I wish to use DPM++ 3M SDE (and the 2M version). You said in the other thread that 3M isn't available yet for the diffusers and the difference is "near-trivial." - still some models suggest it and the composition does change between 2M and 3M. If I want DPM++ 2M SDE, do I select "DPM++ 2M" or "DPM SDE"? and in what combination with the DPM solver algorithm? Do I select sde-dpmsolver? or sde-dpmsolver++? Your reply in the other thread "SDE is a solver type, select it via settings -> samplers -> dpm solver algorithm." This doesn't specific and It's really unclear to me.

[vladmandic]

Adetailer is just one thing but the ability to use ControlNet in a latent second pass is very important to me so it looks like I cannot use Diffusers backend because of this. Please consider if you can adding this functionality back.

Short answer? No.
Long answer? I'm looking how to create optimized workflows, not how to revive old ones just because we're used to them.
So if you tell me what you want to achieve and if that is not possible today, lets look how to design it most elegantly.
If you just want to use image -> controlnet -> hires output, new control does that easily and without the need for second pass. Just use resize menu and set resize option to "before", so resizing happens before final render - isn't that what second pass does?

If I want DPM++ 2M SDE

"DPM++ 2M SDE" never existed as actual sampler. that is basically a preset that was popularized over time.
and nowadays we have soooo many more options, so having preset for every combination of options would be a suicide.
and instead asking which options you need to select to have identical results, why not pick options that work best for you?
again, there is so much more to try.

you keep asking for DPM 3M - do you have a reason? i mean, actual reason other than "I've seen preset used and I want it exactly like that"?

[gohan2091]

Adetailer is just one thing but the ability to use ControlNet in a latent second pass is very important to me so it looks like I cannot use Diffusers backend because of this. Please consider if you can adding this functionality back.

Short answer? No. Long answer? I'm looking how to create optimized workflows, not how to revive old ones just because we're used to them. So if you tell me what you want to achieve and if that is not possible today, lets look how to design it most elegantly. If you just want to use image -> controlnet -> hires output, new control does that easily and without the need for second pass. Just use resize menu and set resize option to "before", so resizing happens before final render - isn't that what second pass does?

Yes I like what you're saying here; if things can be done in a better way then let's discuss that. This is why I am using SD.Next and not A1111 because I like that you're constantly trying to find ways to improve and develop the processes to improve user experience and the diffusers backend is very encouraging but there is some functionality removed which I would love to have included if possible.

I am not sure how to explain my workflow better than I did above but I shall try. Once I find a nice prompt, I like to generation images at 512x768 (or vice versa) and apply a latent x2 upscale as part of the generation. This increases generation time but I like the results. It's nice to generate a batch of 20 images while you make a cup of tea and come back and you have 20 images at 1536 x 1024 at a pretty much automated process. 90% of the time I have Adetailer active as well. It's not super common but I like to load a control net or two such as OpenPose or depth and play around with the stop step and the strength so when those 1536x1024 images are generating, I have a bit of control over the shape of the image. Depth and open pose help shape the images into the direction I want. This works fine in the original backend because the control net is included below the generation settings and I just have to activate it. As a separate control tab, I cannot do that as there is no latent upscaler in the options. The latent upscaler is just superior in the first initial generation to all other upscalers.

If I want DPM++ 2M SDE

"DPM++ 2M SDE" never existed as actual sampler. that is basically a preset that was popularized over time. and nowadays we have soooo many more options, so having preset for every combination of options would be a suicide. and instead asking which options you need to select to have identical results, why not pick options that work best for you? again, there is so much more to try.

you keep asking for DPM 3M - do you have a reason? i mean, actual reason other than "I've seen preset used and I want it exactly like that"?

If I see an image on CivitAI that I want to copy exactly, it's nice to just copy the settings over. The seed, the model, the CFG and the sampler used. It's not super important but it would be nice to have. Why not have things like they are now but in the options, give users the option of ticking additional combinations of presets if they want them? That way it's the best of both worlds? I don't know what options work best for me. I am very confused over the different DPM solver algorithms in the options and what I need to select as a sampler. I just like the option of having the sampler presets that other UIs have. This is not a big deal though and the issue above is much more important to me.

[vladmandic]

dev branch now has a new code that allows resize before AND after. and resize-before can be standard or latent.
that way you can do stuff like: input->upscale/downscale->generate->upscale/downscale->output

[gohan2091]

Thanks, I appreciate your efforts and thank you. I don't know how to use the dev branch so I will wait for the official release so I can play around. I must be honest... I am really disliking the Control menu at the moment and the reason why is because I just don't understand how to use it. Is there a guide going over common workflows that I can read? I've seen some cool things being done with the control menu. For example, you showed an example on DIscord of outpainting.. great but no idea how you did that.

Right now, I am loading a stickman pose, selecting OpenPose for the model and None for processor and I am setting the resolution to 512x768. I've written a simple prompt. Forgetting the latent upscaler for the moment, how do I get it to do a second pass so the scale is x2? There is a "resize by" option and I can select x2 there but where is it getting the initial image dimensions from? It can't be the "resize to" numbers as whatever I do, the image is always 512x768. Setting the "resize to" to "1024x1536" is doing a first pass and is not a suitable resolution for the 1.5 models. Or does your dev mode code address this?

[vladmandic]

A) dev addresses that
B) you're absolutely right we need better docs.
I've just reiterated that to the team as I cannot do everything myself.

[gohan2091]

Having better documentation I feel is really important. It seems you and your team have spent countless hours and a lot of work on SD.Next, it is feature rich and has a lot of potential above and beyond A1111 and other UIs. What good are all these features if some users such as myself do not know how to use them? I am not an expert with AI but I am not a newbie either. I very much welcome better documentation in the future that's easy to understand but goes into specific settings with examples. A video tutorial on Control and other areas for example would be absolutely awesome and I think many people would really appreciate and welcome that but text format with pictures also works.. Thank you for reading my message and I look forward with excitement.

[iDeNoh]

We are working on this as we speak

[gohan2091]

So the latent upscaler is now on Control menu. How do I use it? I load an image that's 512x512 for example, I set a Canny, I go to the before tab, enter 512x512, I then go to scale, select latent (nearest) and enter x2, I then hit generate expecting a 1024x1024 image based on my canny yet the image generated is 512x512 :/ Yes I read https://github.com/vladmandic/automatic/wiki/Control and still cannot figure this out.

[lbeltrame]

It's the "before" you need to set, not the after. After is done at the end.

[gohan2091]

I do have it on "before", I stated that in my message. Please can you re-read?

[lbeltrame]

I need to go back to my notes to confirm, though, as I've been running a series of experiments. How did you run the workflow? Control + independent image? Or Control only? At least in dev Control + input image works like that (tested with SDXL and Openpose).

[gohan2091]

I'm very frustrated with the Control menu, it's incredibly buggy, I can't get most things I want to do to work. Right now I am trying to put a basic depth mask based on an image that I uploaded and I want to generate a new image based on that depth mask. A zoe or midas, I don't care, I just want it to work! It's throwing various errors, the previews aren't working, sometimes it errors so hard that the console becomes unresponsive and I have to close and reload.

I don't even know exactly how to pair up the model dropdown box with the processor dropdown box. I mean in Controlnet extension (which I have no issues with at all), you select a model and it will filter out the items that aren't going to work with it. For example you select Depth and it will show you Depth midas and Depth Zoe for example. In this Control Menu, it shows ALL options regardless of if it works or not. Don't even know what to pair up with softedge to get it to work....

To get back to your question, it was control only mode with an independant image. Would be cool if we could message each other on Facebook or something and you can show me what you know.

[lbeltrame]

I don't have a social presence save here (and on Matrix - see the other thread I posted on Control). That said, I plan on posting the workflow I use when I have the time.
I was able to use Control with a depth map, but again, I'm using dev and not master, and there are a few things fixed not yet in a released version (I switched so I could experiment). The main difference is that I used SDXL and not SD, but I guess it's not relevant for the perspective of running the pipeline.

[gohan2091]

I am using the dev version currently (Have been for the last few days). I have also played around with XL models too. I just can't for the most time get anything I want to work. Do you agree with what I said about pairing up models with processor? I mean any and all options are available (whether they work or not).

[lbeltrame]

I see that all options are offered, but I went directly for what I had in mind (I agree it might be a problem, it's worth filing an issue for it IMO): my depth map was generated with Midas, so I went for "Depth mid".
But the image I had was already preprocessed. I did however test with Canny and Openpose, and both worked: I went up to 3 CNs (I went to great lengths to make sure each CN was actually applied).

[gohan2091]

Can you explain what you mean by "Depth mid" as I don't have that option in neither the ControlNet model dropdown or the Processor drop down. I am asking how do I use Depth Midas, what is the pairing?