Mix models | Generate images using 1.5 and use sd-xl refiner as second pass

[AndreyRGW]

How?

[Tillerz]

My though is that this is two different pipelines and you would have to switch inbetween, so it's probably not going to work right now?

[vladmandic]

But it does.
Select SD pipeline, load model
Select Sd-xl pipeline, load refiner
Enable second pass.
Run generate.

[zethfoxster]

unfortunately i wish i could help :/ but, if someone is loading sdxl and is in the wrong pipeline we get this error...
Error loading model weights:
C:\WebUi\StableDiffusion_July_redo\automatic\models\Stable-diffusion\BaseModels\sd_xl_refiner_0
.9.safetensors Error(s) in loading state_dict for LatentDiffusion:
and most likely the switching of the pipeline was intentional as i see no reason otherwise....
how about instead of this error, it immediately switches and attempts the next pipeline, if that fails, then it fails completely and switches back to whatever the previous was.
I'm not sure about the inner working of it all, just tossing an idea out there.

[Tillerz]

I'd totally recommand have a separate folder for SDXL and then based on where you load stuff from, it automatically switched the pipeline for you so that the load actually works.

[kopyl]

do sizes have to be 1:1? Meaning i can't use 1024 refiner with 512 image, right?

[vladmandic]

in theory ,yes. results are not guaranteed.

[kopyl]

@vladmandic hm... Thank you very much. I need to try it...

[Tillerz]

Sorry, I am totally confused now. What backend do I enable? I stick to the selected one?

So I start with diffusers backend enabled... then I switch pipeline to normal? Load normal model, switch pipeline to diffusers, load the refiner, then i enable second pass which should then have refiner enabled because I am on diffusers backend?

If this is the process (or similar) then we need a simplification here.

I jusat tried that, and the loading (once again) worked, but then the actual rendering fails. Different error now, though.

AttributeError: 'StableDiffusionPipeline' object has no attribute 'model'
Progress 32.29it/s ██████████████████████████████████████ 100% 20/20 00:00 00:00
08:05:04-494278 ERROR    Exception: 'LatentDiffusion' object has no attribute 'scheduler'
08:05:04-496778 ERROR    Arguments: args=('task(tfevnvrhopn2uis)', 'rabbit', '', [], 20, 0, 0, False, False, 1, 1, 6,
                         6, 0.7, 1, -1.0, -1.0, 0, 0, 0, 512, 512, True, 0.3, 1, 'SwinIR_4x', 20, 0, 0, 0, 1, '', '',
                         [], 0, False, 'MultiDiffusion', False, True, 1024, 1024, 96, 96, 48, 4, 'None', 2, False, 10,
                         1, 1, 64, False, False, False, False, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2,
                         -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2,
                         '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0,
                         False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '',
                         'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False,
                         0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 3072, 192, True, True, True,
                         False, False, 7, 100, 'Constant', 0, 'Constant', 0, 4,
                         <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x0000028EB9033DF0>,
                         <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x0000028EB9031EA0>,
                         <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x0000028EB9032B30>,
                         False, False, 1, False, False, False, 1.1, 1.5, 100, 0.7, False, False, True, False, False, 0,
                         'Gustavosta/MagicPrompt-Stable-Diffusion', '', False, False, 'positive', 'comma', 0, False,
                         False, '', 0, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, False, None, None,
                         False, None, None, False, None, None, False, 50) kwargs={}
08:05:04-543800 ERROR    gradio call: AttributeError
┌───────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────┐
│ X:\automatic\modules\call_queue.py:34 in f                                                   │
│                                                                                                                     │
│    33 │   │   │   try:                                                                                              │
│ >  34 │   │   │   │   res = func(*args, **kwargs)                                                                   │
│    35 │   │   │   │   progress.record_results(id_task, res)                                                         │
│                                                                                                                     │
│ X:\automatic\modules\txt2img.py:65 in txt2img                                                │
│                                                                                                                     │
│   64 │   if processed is None:                                                                                      │
│ > 65 │   │   processed = processing.process_images(p)                                                               │
│   66 │   p.close()                                                                                                  │
│                                                                                                                     │
│                                               ... 3 frames hidden ...                                               │
│                                                                                                                     │
│ X:\automatic\modules\processing_diffusers.py:99 in process_diffusers                         │
│                                                                                                                     │
│    98 │   │                                                                                                         │
│ >  99 │   │   if (not hasattr(shared.sd_refiner.scheduler, 'name')) or (shared.sd_refiner.sche                      │
│   100 │   │   │   sampler = sd_samplers.all_samplers_map.get(p.latent_sampler, None)                                │
│                                                                                                                     │
│ X:\automatic\venv\lib\site-packages\torch\nn\modules\module.py:1614 in __getattr__           │
│                                                                                                                     │
│   1613 │   │   │   │   return modules[name]                                                                         │
│ > 1614 │   │   raise AttributeError("'{}' object has no attribute '{}'".format(                                     │
│   1615 │   │   │   type(self).__name__, name))                                                                      │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
AttributeError: 'LatentDiffusion' object has no attribute 'scheduler'

Maybe it's a new bug now? I get that err even when I switch everything to SDXL/diffusers.

Version: 0a083c24 Tue Jul 18 20:43:30 2023 -0400

[vladmandic]

Set backend to diffusers.

Set pipeline to sd
Load sd model

Set pipeline to sd-xl
Load sd-xl refiner

Enable second pass
Run generate

[josh-robey]

These options are all gone in my settings, I'm assuming cause of updates. How would someone do this now?

[amauri-di-maia]

True, lets see if someone give us some tips about it!

[vladmandic]

they are still there. and replying to discussion which is 7 months old is not how to get results.