Many thanks to all open-source contributors to and its dependencies.

This repository contains code to replicate the paper "Differentiable Time-Frequency Scattering on GPU" (published at DAFx 2022, best paper award).

Time-frequency scattering is available in in beta and will be released in v0.4. To use this implementation, install Kymatio from source. To replicate the results in this paper, follow the installation instructions below.

We assess Time-Frequency Scattering in Kymatio for 3 machine listening research applications:

• unsupervised manifold learning of spectrotemporal modulations
• hybrid jtfs + convnet supervised musical instrument classification
• texture resynthesis

First, install dependencies

# clone project   
git clone https://github.com/cyrusvahidi/jtfs-gpu

# install project   
cd jtfs-gpu
pip install -e .
pip install -r requirements.txt

# install kymatio from source
cd kymatio
python setup.py develop

• The JTFS algorithm source code to replicate the paper can be found
• The latest version of JTFS can be installed directly from the source

Download Medley-solos-DB

python scripts/process_msdb_features.py --data_dir <your_msdb_dir> --feature <feature_to_extract>

In /scripts/gin/config.gin set MSDB_DATA_DIR and MSDB_CSV according to the absolute path of your MSDB download.

python scripts/train_cnn.py

python scripts/isomap.py

see the output in /img

cd notebooks
jupyter notebook

See Scale-Rate Visualization.ipynb and Resynthesis results.ipynb

H E L L O

• Factors of variation:
• $f_c$ carrier frequency
• $f_m$ amplitude modulation frequency
• $\gamma$ chirp rate

• MFCCs
• Time Scattering
• Time-Frequency Scattering
• Open-L3
• Spectrotemporal Receptive Field (STRF)

@article{muradeli2022differentiable,
  title={Differentiable Time-Frequency Scattering in Kymatio},
  author={John Muradeli, Cyrus Vahidi, Changhong Wang, Han Han, Vincent Lostanlen, Mathieu Lagrange, George Fazekas},
  journal={arXiv preprint arXiv:2204.08269},
  year={2022}
}