relyme

ReLyMe

ReLyMe: Improving Lyric-to-Melody Generation by Incorporating Lyric-Melody Relationships, by Chen Zhang, LuChin Chang, Songruoyao Wu, Xu Tan, Tao Qin, Tie-Yan Liu, Kejun Zhang, ACM Multimedia 2022, is a method that leverages lyric-melody relationships from music theory to alleviate the dissonance between lyrics and melodies. Specifically, we first introduce several principles that lyrics and melodies should follow in terms of tone, rhythm, and structure relationships according to musicians and composers. These principles are then integrated into neural network based lyric-to-melody models by adding corresponding constraints during the decoding process to improve the harmony between lyrics and melodies. We further design a series of objective and subjective metrics to evaluate the generated melodies. Experiments on both English and Chinese song datasets show the effectiveness of ReLyMe, demonstrating the superiority of leveraging the principles (lyric-melody relationships) from music domain for neural based lyric-to-melody generation.

relyme.mp4

The demo video of ReLyMe.

1 Environment

All python packages used in ReLyMe are listed in requirements.txt

pip install -r requirements.txt

2 Traning

For the training details of TeleMelody and SongMASS, please refer to https://github.com/microsoft/muzic/tree/main/telemelody and https://github.com/microsoft/muzic/tree/main/songmass.

3 ReLyMe based on TeleMelody

To use ReLyMe in TeleMelody, please first follow the instructions here, and have it run successfully on your machine. After having TeleMelody work successfully, please follow the steps below:

1. Copy all the files under ReLyMe/telemelody_zh or ReLyMe/telemelody_en to YOUR_TELEMELODY_PATH/inferrence
2. Then, replace sequence_generator.py and fairseq_task.py in Fairseq packages with the ones we provide in ReLyMe/telemelody_zh or ReLyMe/telemelody_en.

cd [YOUR PYTHON INTERPRETER PATH]/lib/python3.7/site-packages/fairseq
mv sequence_generator.py sequence_generator_bk.py
mv fairseq_task.py fairseq_task_bk.py
cp sequence_generator.py [YOUR PYTHON INTERPRETER PATH]/lib/python3.7/site-packages/fairseq/
cp fairseq_task.py [YOUR PYTHON INTERPRETER PATH]/lib/python3.7/site-packages/fairseq/tasks

3. Set the global variable "GEN_MODE" in config.py to "BASE" for generating TeleMelody baseline output, and "ReLyMe" for ReLyMe output.

import numpy as np
from enum import Enum
from midi_utils import number_to_note

GEN_MODE = "BASE"
# GEN_MODE = "ReLyMe"

4. Finally, run the main.py

python main.py

4 ReLyMe based on SongMASS

(TBD)

5 Evaluation

We provide a score module (ReLyMe/score) to perform objective evaluation. For Chinese, you should prepare {zh_song_prefix}.mid (midi files) and {zh_song_prefix}.strct (structure file) in the same directory. For English, you should prepare {en_song_prefix}.mid (midi files), {en_song_prefix}.strct (structure file) {en_song_prefix}.syl (syllable file) in the same directory.

We provide sample files under ReLyMe/score/testmid.

score/testmid
├── en
│   ├── tele-en.mid
│   ├── tele-en.strct
│   └── tele-en.syl
└── zh
    ├── tele-zh.mid
    └── tele-zh.strct

There is two way you can use the score module.

1. Use the score_zh.py or score_en.py directly: Set the song_prefix to the one you want at the final lines in score_zh.py or score_en.py. And then run score_zh.py or score_en.py.

if __name__ == "__main__":
	score = get_score("testmid/zh/tele-zh")

python score_en.py

2. Import the ReLyMe/score as an module: Import module and use the score.get_score() to get the score of song_prefix

import os
import sys
sys.path.append(os.path.join({PATH_TO_ReLyMe/score}))
import score.score_en as score_en

print(score_en.get_score(song_prefix))