I managed to hose my SDNext install ... somehow ... and can't fix it

[afbagwell]

I was poking around with pip and trying to upgrade some libraries related to CUDA. It didn't work, so I attempted to back out my changes. Everything appeared to work fine again, but then I ran into a brick wall.

Every time I try to use hires-fix to upscale anything larger than 512x512, the process drastically chews up GPU memory and slows down. In the screen shot below, the first run was a single 512x512 image with the second-pass Lanczos upscaler set to double the image size to 1024x0124. The second one is the exact same, but with the initial image size set to 640x640. All my added plugins are turned off in both cases.

It should be noted that while the second image says only 4.69 out of 24G of memory was used, my Windows Task Manager showed the GPU running at full 24G capacity during the entire 14 minutes it took to complete the image.

Here's what I've tried so far that hasn't worked.

1. doing an in-place SDNext reinstall through the command line
2. removing the venv folder and letting SDNext completely rebuild it
3. installing a completely new instance of SDNext and running it basically out-of-the-box

The last one really gob-smacked me because a fresh install with no connection to the other instance exhibited the same behavior. So to that end I tried:

1. Re-installing my base python install.
2. Re-installing the Nvidia CUDA toolkit for windows
3. And of course, turning my machine off and back on again

None of which helped.

I also have A1111 installed on my machine, so I tried it with the exact same parameters and it does not encounter the upscaling slowdown; the 640x640 image upscaling only took a few seconds. So somehow, this is unique to running SDNext.

Has anyone seen something like this happen or have a suggestion as to what I can do to fix it?

For ref, here's my startup settings. I'm running with a 4090.

Using VENV: D:\AiApps\StableDiffusion\sdnext15\venv
19:20:34-949326 INFO     Starting SD.Next
19:20:34-952326 INFO     Logger: file="D:\AiApps\StableDiffusion\sdnext15\sdnext.log" level=INFO size=24389121
                         mode=append
19:20:34-953326 INFO     Python 3.10.13 on Windows
19:20:35-051328 INFO     Version: app=sd.next updated=2024-01-30 hash=9bf08c54
                         [email protected]:vladmandic/automatic.git/tree/master
19:20:35-579326 INFO     Platform: arch=AMD64 cpu=Intel64 Family 6 Model 183 Stepping 1, GenuineIntel system=Windows
                         release=Windows-10-10.0.22631-SP0 python=3.10.13
19:20:35-582326 INFO     nVidia CUDA toolkit detected: nvidia-smi present
19:20:35-615326 WARNING  Modified files: ['scripts/detect_extension.py']
19:20:35-690326 INFO     Extensions: disabled=[]
19:20:35-691326 INFO     Extensions: enabled=['Lora', 'sd-extension-chainner', 'sd-extension-system-info',
                         'sd-webui-agent-scheduler', 'sd-webui-controlnet', 'stable-diffusion-webui-images-browser',
                         'stable-diffusion-webui-rembg'] extensions-builtin
19:20:35-696326 INFO     Extensions: enabled=['adetailer', 'canvas-zoom', 'sd-dynamic-prompts',
                         'sd-dynamic-thresholding', 'sd-webui-aspect-ratio-helper', 'sd-webui-faceswaplab',
                         'sd-webui-inpaint-anything', 'sd-webui-openpose-editor', 'sd-webui-reactor',
                         'sd-webui-regional-prompter', 'sd-webui-segment-anything', 'sd-webui-supermerger',
                         'ultimate-upscale-for-automatic1111'] extensions
19:20:35-704326 INFO     Startup: quick launch
19:20:35-704326 INFO     Verifying requirements
19:20:35-715326 INFO     Verifying packages
19:20:35-717326 INFO     Extensions: disabled=[]
19:20:35-717326 INFO     Extensions: enabled=['Lora', 'sd-extension-chainner', 'sd-extension-system-info',
                         'sd-webui-agent-scheduler', 'sd-webui-controlnet', 'stable-diffusion-webui-images-browser',
                         'stable-diffusion-webui-rembg'] extensions-builtin
19:20:35-720326 INFO     Extensions: enabled=['adetailer', 'canvas-zoom', 'sd-dynamic-prompts',
                         'sd-dynamic-thresholding', 'sd-webui-aspect-ratio-helper', 'sd-webui-faceswaplab',
                         'sd-webui-inpaint-anything', 'sd-webui-openpose-editor', 'sd-webui-reactor',
                         'sd-webui-regional-prompter', 'sd-webui-segment-anything', 'sd-webui-supermerger',
                         'ultimate-upscale-for-automatic1111'] extensions
19:20:35-728326 INFO     Extension preload: {'extensions-builtin': 0.0, 'extensions': 0.0}
19:20:35-730326 INFO     Command line args: []
19:20:39-422339 INFO     Load packages: torch=2.2.0+cu121 diffusers=0.25.0 gradio=3.43.2
19:20:39-892340 INFO     Engine: backend=Backend.ORIGINAL compute=cuda mode=no_grad device=cuda
                         cross-optimization="Scaled-Dot-Product"
19:20:39-937340 INFO     Device: device=NVIDIA GeForce RTX 4090 n=1 arch=sm_90 cap=(8, 9) cuda=12.1 cudnn=8801
                         driver=551.23
19:20:41-990847 INFO     Available VAEs: path="D:\AiApps\StableDiffusion\sd15-models\VAE" items=3
19:20:41-992847 INFO     Disabled extensions: []
19:20:41-998847 INFO     Available models: path="D:\AiApps\StableDiffusion\a1111\models\Stable-diffusion" items=83
                         time=0.00
19:20:42-537847 INFO     Extension: script='scripts\faceid.py' [2;36m19:20:42-536846[0m[2;36m [0m[1;31mERROR   [0m
                         FaceID: No module named [32m'ip_adapter'[0m
19:20:43-253847 INFO     Extension: script='extensions-builtin\sd-webui-agent-scheduler\scripts\task_scheduler.py' Using
                         sqlite file: extensions-builtin\sd-webui-agent-scheduler\task_scheduler.sqlite3
19:20:43-372847 INFO     Extension: script='extensions-builtin\sd-webui-controlnet\scripts\api.py' ControlNet
                         preprocessor location:
                         D:\AiApps\StableDiffusion\sdnext15\extensions-builtin\sd-webui-controlnet\annotator\downloads
19:20:43-440847 INFO     Extension: script='extensions-builtin\sd-webui-controlnet\scripts\controlnet.py' Warning:
                         ControlNet failed to load SGM - will use LDM instead.
19:20:43-452847 INFO     Extension: script='extensions-builtin\sd-webui-controlnet\scripts\hook.py' Warning: ControlNet
                         failed to load SGM - will use LDM instead.
19:20:44-439847 INFO     Extension: script='extensions\adetailer\scripts\!adetailer.py' [-] ADetailer initialized.
                         version: 24.1.2, num models: 14
19:20:44-486847 INFO     Extension: script='extensions\sd-dynamic-thresholding\scripts\dynamic_thresholding.py' ======
19:20:44-487847 INFO     Extension: script='extensions\sd-dynamic-thresholding\scripts\dynamic_thresholding.py' Warning!
                         Using legacy KDiff version! Is your WebUI up to date?
19:20:44-488847 INFO     Extension: script='extensions\sd-dynamic-thresholding\scripts\dynamic_thresholding.py' ======
19:20:44-769847 INFO     Extension: script='extensions\sd-webui-inpaint-anything\scripts\inpaint_anything.py' Creating
                         custom formatter for non UTF-8 environments due to attribute 'encoding' of '_io._TextIOBase'
                         objects is not writable
19:20:44-954352 INFO     Extensions time: 2.79 { sdnext15=0.38 Lora=0.41 sd-webui-agent-scheduler=0.24
                         sd-webui-controlnet=0.23 stable-diffusion-webui-images-browser=0.17 adetailer=0.79
                         sd-webui-faceswaplab=0.07 sd-webui-inpaint-anything=0.18 sd-webui-segment-anything=0.08 }
19:20:45-055353 INFO     Load UI theme: name="black-teal" style=Dark base=sdnext.css
19:20:48-097860 INFO     Local URL: http://127.0.0.1:7860/
19:20:48-098861 INFO     Initializing middleware
19:20:48-248860 INFO     [AgentScheduler] Task queue is empty
19:20:48-250860 INFO     [AgentScheduler] Registering APIs
19:20:48-499860 INFO     Startup time: 12.77 { torch=2.78 gradio=0.88 libraries=2.57 extensions=2.79 face-restore=0.17
                         extra-networks=0.06 ui-extra-networks=0.06 ui-txt2img=0.14 ui-img2img=0.16 ui-settings=0.35
                         ui-extensions=1.66 ui-defaults=0.08 launch=0.53 api=0.06 app-started=0.33 }
19:20:54-496827 INFO     MOTD: N/A
19:21:01-020872 INFO     Browser session: user=None client=127.0.0.1 agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64;
                         rv:122.0) Gecko/20100101 Firefox/122.0

[afbagwell]

On a lark I decided to downgrade my Nvidia driver to 531. I ran the same simple 768x768 generation again with upscaling to twice the size and got this error.

So yes, somehow I managed to bork CUDA at least where SDNext is concerned. My A1111 instance performed fine (even slightly faster) with the older driver.

100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:03<00:00,  5.61it/s]
  0%|                                                                                                                                                                             | 0/20 [00:02<?, ?it/s]
20:53:51-080099 ERROR    Exception: CUDA out of memory. Tried to allocate 40.50 GiB. GPU 0 has a total capacity of 23.99 GiB of which 18.77 GiB is free. Of the allocated memory 3.55 GiB is allocated by
                         PyTorch, and 42.43 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid
                         fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
20:53:51-082099 ERROR    Arguments: args=('task(5ge0p88bdwde0lj)', 'abstract art', '', [], 20, 2, 3, True, True, False, 12, 1, 7, 4, 0.7, 1, 3490647201.0, -1.0, 0, 0, 0, 768, 768, True, 0.4, 2,
                         'Lanczos', True, 20, 0, 0, 5, 0.8, '', '', False, 4, 0.95, False, 1, 1, False, 0.6, 1, [], 0, False, False, 'positive', 'comma', 0, False, False, '', 0, '', [], 0, '', [], 0,
                         '', [], False, True, False, False, False, False, 0, None, None, False, None, None, False, None, None, False, 50, [], 30, '', 4, [], 1, '', '', '', '',
                         UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1, threshold_a=-1,
                         threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=False, hr_option='Both', save_detected_map=True,
                         advanced_weighting=None), UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize', low_vram=False, processor_res=-1,
                         threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=False, hr_option='Both',
                         save_detected_map=True, advanced_weighting=None), UiControlNetUnit(enabled=False, module='none', model='None', weight=1, image=None, resize_mode='Crop and Resize',
                         low_vram=False, processor_res=-1, threshold_a=-1, threshold_b=-1, guidance_start=0, guidance_end=1, pixel_perfect=False, control_mode='Balanced', inpaint_crop_input_image=False,
                         hr_option='Both', save_detected_map=True, advanced_weighting=None), False, False, {'ad_model': 'face_yolov8m.pt', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence':
                         0.7, 'ad_mask_k_largest': 0, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 4, 'ad_mask_merge_invert': 'None',
                         'ad_mask_blur': 4, 'ad_denoising_strength': 0.3, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width':
                         512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_checkpoint': False, 'ad_checkpoint': 'Use same
                         checkpoint', 'ad_use_vae': False, 'ad_vae': 'Use same VAE', 'ad_use_sampler': False, 'ad_sampler': 'UniPC', 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1,
                         'ad_use_clip_skip': False, 'ad_clip_skip': 1, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_module': 'None', 'ad_controlnet_weight': 1,
                         'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, {'ad_model': 'hand_yolov8s.pt', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence':
                         0.7, 'ad_mask_k_largest': 0, 'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 4, 'ad_mask_merge_invert': 'None',
                         'ad_mask_blur': 4, 'ad_denoising_strength': 0.3, 'ad_inpaint_only_masked': True, 'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width':
                         512, 'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7, 'ad_use_checkpoint': False, 'ad_checkpoint': 'Use same
                         checkpoint', 'ad_use_vae': False, 'ad_vae': 'Use same VAE', 'ad_use_sampler': False, 'ad_sampler': 'UniPC', 'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1,
                         'ad_use_clip_skip': False, 'ad_clip_skip': 1, 'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_module': 'None', 'ad_controlnet_weight': 1,
                         'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, False, False, 1, False, False, False, 1.1, 1.5, 100, 0.7, False, False, True, False, False, 0,
                         'Gustavosta/MagicPrompt-Stable-Diffusion', '', False, 9, 98.5, 'Half Cosine Up', 3.5, 'Half Cosine Up', 3.5, 4, True, 'MEAN', 'AD', 1, None, False, '0', '0',
                         'inswapper_128.onnx', 'CodeFormer', 1, True, 'None', 1, 1, False, True, 1, 0, 0, False, 0.5, True, False, 'CUDA', False, 0, 'None', '', None, False, False, False, False,
                         'Matrix', 'Columns', 'Mask', 'Prompt', '1,1', '0.2', False, False, False, 'Attention', [False], '0', '0', '0.4', None, '0', '0', False, False, False, 0, None, [], 0, False, [],
                         [], False, 0, 1, False, False, 0, None, [], -2, False, [], False, 0, None, None) kwargs={}
20:53:51-090098 ERROR    gradio call: OutOfMemoryError
╭───────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────────────────────────────────────────────╮
│ D:\AiApps\StableDiffusion\sdnext15\modules\call_queue.py:31 in f                                                                                                                                     │
│                                                                                                                                                                                                      │
│   30 │   │   │   try:                                                                                                                                                                                │
│ ❱ 31 │   │   │   │   res = func(*args, **kwargs)                                                                                                                                                     │
│   32 │   │   │   │   progress.record_results(id_task, res)                                                                                                                                           │
│                                                                                                                                                                                                      │
│ D:\AiApps\StableDiffusion\sdnext15\modules\txt2img.py:88 in txt2img                                                                                                                                  │
│                                                                                                                                                                                                      │
│   87 │   if processed is None:                                                                                                                                                                       │
│ ❱ 88 │   │   processed = processing.process_images(p)                                                                                                                                                │
│   89 │   p.close()                                                                                                                                                                                   │
│                                                                                                                                                                                                      │
│ D:\AiApps\StableDiffusion\sdnext15\modules\processing.py:760 in process_images                                                                                                                       │
│                                                                                                                                                                                                      │
│    759 │   │   │   with context_hypertile_vae(p), context_hypertile_unet(p):                                                                                                                         │
│ ❱  760 │   │   │   │   res = process_images_inner(p)                                                                                                                                                 │
│    761                                                                                                                                                                                               │
│                                                                                                                                                                                                      │
│ D:\AiApps\StableDiffusion\sdnext15\extensions-builtin\sd-webui-controlnet\scripts\batch_hijack.py:41 in processing_process_images_hijack                                                             │
│                                                                                                                                                                                                      │
│    40 │   │   │   # we are not in batch mode, fallback to original function                                                                                                                          │
│ ❱  41 │   │   │   return getattr(processing, '__controlnet_original_process_images_inner')(p,                                                                                                        │
│    42                                                                                                                                                                                                │
│                                                                                                                                                                                                      │
│ D:\AiApps\StableDiffusion\sdnext15\modules\processing.py:900 in process_images_inner                                                                                                                 │
│                                                                                                                                                                                                      │
│    899 │   │   │   │   with devices.without_autocast() if devices.unet_needs_upcast else device                                                                                                      │
│ ❱  900 │   │   │   │   │   samples_ddim = p.sample(conditioning=c, unconditional_conditioning=u                                                                                                      │
│    901 │   │   │   │   x_samples_ddim = [decode_first_stage(p.sd_model, samples_ddim[i:i+1].to(                                                                                                      │
│                                                                                                                                                                                                      │
│                                                                                       ... 34 frames hidden ...                                                                                       │
│                                                                                                                                                                                                      │
│ D:\AiApps\StableDiffusion\sdnext15\repositories\ldm\modules\attention.py:272 in _forward                                                                                                             │
│                                                                                                                                                                                                      │
│   271 │   def _forward(self, x, context=None):                                                                                                                                                       │
│ ❱ 272 │   │   x = self.attn1(self.norm1(x), context=context if self.disable_self_attn else Non                                                                                                       │
│   273 │   │   x = self.attn2(self.norm2(x), context=context) + x                                                                                                                                     │
│                                                                                                                                                                                                      │
│ D:\AiApps\StableDiffusion\sdnext15\venv\lib\site-packages\torch\nn\modules\module.py:1511 in _wrapped_call_impl                                                                                      │
│                                                                                                                                                                                                      │
│   1510 │   │   else:                                                                                                                                                                                 │
│ ❱ 1511 │   │   │   return self._call_impl(*args, **kwargs)                                                                                                                                           │
│   1512                                                                                                                                                                                               │
│                                                                                                                                                                                                      │
│ D:\AiApps\StableDiffusion\sdnext15\venv\lib\site-packages\torch\nn\modules\module.py:1520 in _call_impl                                                                                              │
│                                                                                                                                                                                                      │
│   1519 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                                                                                                                       │
│ ❱ 1520 │   │   │   return forward_call(*args, **kwargs)                                                                                                                                              │
│   1521                                                                                                                                                                                               │
│                                                                                                                                                                                                      │
│ D:\AiApps\StableDiffusion\sdnext15\modules\sd_hijack_optimizations.py:395 in scaled_dot_product_no_mem_attention_forward                                                                             │
│                                                                                                                                                                                                      │
│   394 │   with torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=True, enable_mem_                                                                                                       │
│ ❱ 395 │   │   return scaled_dot_product_attention_forward(self, x, context, mask)                                                                                                                    │
│   396                                                                                                                                                                                                │
│                                                                                                                                                                                                      │
│ D:\AiApps\StableDiffusion\sdnext15\modules\sd_hijack_optimizations.py:380 in scaled_dot_product_attention_forward                                                                                    │
│                                                                                                                                                                                                      │
│   379 │   # the output of sdp = (batch, num_heads, seq_len, head_dim)                                                                                                                                │
│ ❱ 380 │   hidden_states = torch.nn.functional.scaled_dot_product_attention(                                                                                                                          │
│   381 │   │   q, k, v, attn_mask=mask, dropout_p=0.0, is_causal=False                                                                                                                                │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
OutOfMemoryError: CUDA out of memory. Tried to allocate 40.50 GiB. GPU 0 has a total capacity of 23.99 GiB of which 18.77 GiB is free. Of the allocated memory 3.55 GiB is allocated by PyTorch, and 42.43
MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory
Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)