DMC

Code for paper Deep Multimodal Clustering for Unsupervised Audiovisual Learning. The code is implemented on PyTorch with python3. This repo includes training and testing code for audiovisual clustering with unlabeled videos.

Requirements

• PyTorch 1.1
• torchvision
• scikit-learn
• librosa
• Pillow

Running Procedure

Data Preparation

• Download dataset, e.g., MUSIC, and split into training/validation/testing set

• Extract frames at 4 fps by running

  python3 data/cut_video.py
  

• Extract 1-second audio clips and turn into Log-Mel-Spectrogram by running

  python3 data/cut_audio.py
  

Training

location_dmc.py [-h]
optional arguments:
[--batch_size] training batchsize
[--learning_rate] learning rate
[--epoch] total training epoch
[--evaluate] only do testing or also training
[--v_cluster] number of visual clusters
[--a_cluster] number of audio clusters

python3 location_dmc.py --v_cluster 2 --a_cluster 1

Normally, the number of audio clusters is set to the number of audio classes, e.g., 1 for solo videos, and the number of visual clusters is set to the number of visual object classes plus 1 (background), e.g., 2 for solo videos.

Testing

python3 location_dmc.py --v_cluster 2 --a_cluster 1 --evaluate 1 --use_pretrain 1 --ckpt_file path/to/ckpt

The clustering result is saved in dmc.pkl, it could be used for qualitative visualization as well as quantitative evaluation.

Examples

• Solo videos

• Duet videos

Citation

@inproceedings{hu2019deep,
  title={Deep Multimodal Clustering for Unsupervised Audiovisual Learning},
  author={Hu, Di and Nie, Feiping and Li, Xuelong},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={9248--9257},
  year={2019}
}

@article{hu2020curriculum,
  title={Curriculum audiovisual learning},
  author={Hu, Di and Wang, Zheng and Xiong, Haoyi and Wang, Dong and Nie, Feiping and Dou, Dejing},
  journal={arXiv preprint arXiv:2001.09414},
  year={2020}
}