Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MM] Install and load Stable Diffusion 3 models #6512

Open
wants to merge 21 commits into
base: main
Choose a base branch
from

Conversation

lstein
Copy link
Collaborator

@lstein lstein commented Jun 13, 2024

Summary

This implements alpha-level support for generating images with Stable Diffusion 3.

To use:

  1. Install the Diffusers version of SD3, located at stabilityai/stable-diffusion-3-medium-diffusers. For convenience, this repo has been added to the Starter Models pane of the model manager UI.
  2. Create a workflow like the one shown below:
    image
  3. If you are on a CUDA system with less than 20 GB of VRAM, deactivate the "Use TE5 Encoder" slider in the Stable Diffusion 3 node. Otherwise you will run out of memory!

Related Issues / Discussions

Currently only diffusers models can be loaded. Upstream support for loading checkpoints is a WIP.

  • Probe SD3 diffusers
  • Probe SD3 checkpoints
  • Load SD3 diffusers
  • Load SD3 checkpoints
  • Load SD3 diffusers models with a quantized text_encoder_3
  • Unload SD3 diffusers models with a quantized text_encoder_3 (not confirmed working)
  • Download and install SD3 diffusers
  • Download and install SD3 checkpoints

Models using a Quantized text_encoder_3

There is preliminary support for such models, but they still do not fit into a 12 GB VRAM machine and this feature has not been tested. Systems with adequate VRAM may be able to load and run these models, but be aware that there is an unsolved issue with an 8K VRAM memory leak that occurs with each generation.

Note that a quantized model will only run on CUDA systems.

QA Instructions

Follow the instructions above.

Merge Plan

Squash when approved.

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • Documentation added / updated (if applicable)

@github-actions github-actions bot added python PRs that change python files Root backend PRs that change backend files python-deps PRs that change python dependencies labels Jun 13, 2024
@github-actions github-actions bot added the frontend PRs that change frontend files label Jun 14, 2024
@github-actions github-actions bot added the invocations PRs that change invocations label Jun 14, 2024
@github-actions github-actions bot added the services PRs that change app services label Jun 16, 2024
@lstein lstein marked this pull request as ready for review June 20, 2024 22:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend PRs that change backend files frontend PRs that change frontend files invocations PRs that change invocations python PRs that change python files python-deps PRs that change python dependencies Root services PRs that change app services
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants