Differentiable Time-Frequency Scattering on GPU 🌊

source code by

Cyrus Vahidi²
Changhong Wang¹, Han Han¹ Vincent Lostanlen¹
John Muradeli
LS2N/CNRS Centrale Nantes¹ Queen Mary University of London²

Many thanks to all open-source contributors to and its dependencies.

This repository contains code to replicate the paper "Differentiable Time-Frequency Scattering on GPU" (published at DAFx 2022, best paper award).

Time-frequency scattering is available in in beta and will be released in v0.4. To use this implementation, install Kymatio from source. To replicate the results in this paper, follow the installation instructions below.

We assess Time-Frequency Scattering in Kymatio for 3 machine listening research applications:

• unsupervised manifold learning of spectrotemporal modulations
• hybrid jtfs + convnet supervised musical instrument classification
• texture resynthesis

How to run

First, install dependencies

Installation

# clone project   
git clone https://github.com/cyrusvahidi/jtfs-gpu

# install project   
cd jtfs-gpu
pip install -e .
pip install -r requirements.txt

# install kymatio from source
cd kymatio
python setup.py develop

• The JTFS algorithm source code to replicate the paper can be found
• The latest version of JTFS can be installed directly from the source

ConvNet Classifier

Download Medley-solos-DB

Extract Scattering Features

python scripts/process_msdb_features.py --data_dir <your_msdb_dir> --feature <feature_to_extract>

Configure gin

In /scripts/gin/config.gin set MSDB_DATA_DIR and MSDB_CSV according to the absolute path of your MSDB download.

Run training

python scripts/train_cnn.py

Isomap Visualizations & K-NN Regression

python scripts/isomap.py

see the output in /img

Scale-Rate Visualizations and Resynthesis

cd notebooks
jupyter notebook

See Scale-Rate Visualization.ipynb and Resynthesis results.ipynb

Notebook Guide

Scale-Rate Visualisations

H E L L O

Synthetic amplitude-modulated chirp dataset

• Factors of variation:
• $f_c$ carrier frequency
• $f_m$ amplitude modulation frequency
• $\gamma$ chirp rate

Manifold Embedding of the Nearest Neighbour Graph

• MFCCs
• Time Scattering
• Time-Frequency Scattering
• Open-L3
• Spectrotemporal Receptive Field (STRF)

K-NN regression of synthesizer parameters

2-D CNN classifier

Differentiable Resynthesis

Citation

@article{muradeli2022differentiable,
  title={Differentiable Time-Frequency Scattering in Kymatio},
  author={John Muradeli, Cyrus Vahidi, Changhong Wang, Han Han, Vincent Lostanlen, Mathieu Lagrange, George Fazekas},
  journal={arXiv preprint arXiv:2204.08269},
  year={2022}
}