Riffusion v0.3

Riffusion is a library for real-time music and audio generation with stable diffusion.

Read about it at https://www.riffusion.com/about and try it at https://www.riffusion.com/.

🖊️ Full Rewrite

This release contains a full rewrite of the Riffusion codebase to go from a hack to a quality software project.

• Rename the repository from riffusion-inference to riffusion.
• SpectrogramParams class that contains all conversion parameters, with sane defaults.
• SpectrogramConverter class that converts between spectrogram tensors and audio.
• SpectrogramImageConverter class that converts between spectrogram images and audio.
• Leverage pydub AudioSegment in more places rather than raw numpy arrays.
• Move common code into the util package.
• Cache more computation and be careful about error checking.
• Move third party integrations into the integrations package. Share most of the code so they greatly simplify.
• pyproject.toml for tool configuration
• Overhaul README with more descriptive instructions.

🚨 This release is API compatible with the web app, but code that used this repository directly will need to be updated.

👩‍💻 Riffusion CLI

Extensible command line interface for performing common tasks. See the README for details.

$ python -m riffusion.cli -h
usage: cli.py [-h] {audio-to-image,image-to-audio,sample-clips,print-exif} ...

positional arguments:
  {audio-to-image,image-to-audio,sample-clips,print-exif}
    audio-to-image      Compute a spectrogram image from a waveform.
    image-to-audio      Reconstruct an audio clip from a spectrogram image.
    sample-clips        Slice an audio file into clips of the given duration.
    print-exif          Print the params of a spectrogram image as saved in the exif data.

options:
  -h, --help            show this help message and exit

🤾‍♂️ Riffusion Playground

Extensible Streamlit app for interactive exploration of Riffusion. See the README for details.

🔥 MPS and CPU Backends

Riffusion now can run on MPS and CPU backends in addition to CUDA. See the README for details.

Also adds graceful detection and fallback of devices.

Closes: #15

👓 Stereo Spectrograms

Add tools to encode and decode stereo audio as spectrograms, using the G and B channels for left and right.

🖼️ Encode Spectrogram Params in Image EXIF

Add the ability to store spectrogram conversion parameters in EXIF metadata of the images, and the ability to decode back to audio from those params. This allows more flexibility for usage without assuming default parameters.

The SpectrogramParams class has methods to convert to and from EXIF.

$ python -m riffusion.cli print-exif --image spectrogram.jpg
NUM_FREQUENCIES      =             512
STEP_SIZE_MS         =              10
MAX_VALUE            =      46801012.0
MIN_FREQUENCY        =               0
WINDOW_DURATION_MS   =             100
MAX_FREQUENCY        =           10000
PADDED_DURATION_MS   =             400
SAMPLE_RATE          =           44100
STEREO               =               1
POWER_FOR_IMAGE      =            0.25

🔉 Post-Processing Filters

Add a capability to apply normalization and compression to audio using pydub.

🟢 Test Suite

Add a suite of tests in the test/ package, and check in some test data.

They are automatically run on pull requests, configured from ci.yml.

• audio_to_image_test.py
• image_to_audio_test.py
• image_util_test.py
• linter_test.py
• print_exif_test.py
• sample_clips_test.py
• spectrogram_converter_test.py
• spectrogram_image_converter_test.py

🧹 Lint Tools

These tools run in CI and must pass cleanly to merge.

• ruff for linting (ruff --fix .)
• black for formatting (black .)
• mypy for typing (mypy .)

PRs

• Rewrite the codebase to be high quality by @hmartiro in #36
• Enable ruff import sorting by @hmartiro in #38
• Add CI with github actions by @hmartiro in #37
• Streamlit app for interactive use of the model by @hmartiro in #40
• Add detail to readme by @hmartiro in #46
• Disable compression by default, too slow by @hmartiro in #47
• Improve interpolation playground by @hmartiro in #45

Full Changelog: v0.2.0...v0.3.0