ESPnet2/Course/CMU_SpeechProcessing_Spring2023/assignment8_tts.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SMSw_r1uRm4a"
      },
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/espnet/notebook/blob/master/espnet2_tts_realtime_demo.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MuhqhYSToxl7"
      },
      "source": [
        "# CMU 11492/11692 Spring 2023: Text to Speech\n",
        "\n",
        "In this demonstration, we will show you some demonstrations of text to speech systems in ESPnet. \n",
        "\n",
        "Main references:\n",
        "- [ESPnet repository](https://github.com/espnet/espnet)\n",
        "- [ESPnet documentation](https://espnet.github.io/espnet/)\n",
        "\n",
        "Author:\n",
        "- Siddhant Arora (siddhana@andrew.cmu.edu)\n",
        "\n",
        "The notebook is adapted from this [Colab](https://colab.research.google.com/github/espnet/notebook/blob/master/espnet2_tts_realtime_demo.ipynb)\n",
        "\n",
        "## ❗Important Notes❗\n",
        "- We are using Colab to show the demo. However, Colab has some constraints on the total GPU runtime. If you use too much GPU time, you may not be able to use GPU for some time.\n",
        "- There are multiple in-class checkpoints ✅ throughout this tutorial. **Your participation points are based on these tasks.** Please try your best to follow all the steps! If you encounter issues, please notify the TAs as soon as possible so that we can make an adjustment for you.\n",
        "- Please submit PDF files of your completed notebooks to Gradescope. You can print the notebook using `File -> Print` in the menu bar."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9e_i_gdgAFNJ"
      },
      "source": [
        "## Installation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fjJ5zkyaoy29"
      },
      "source": [
        "# NOTE: pip shows imcompatible errors due to preinstalled libraries but you do not need to care\n",
        "!pip install typeguard==2.13.3\n",
        "!git clone --depth 5 -b spoken_dialog_demo https://github.com/siddhu001/espnet.git\n",
        "!cd espnet && pip install ."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install parallel_wavegan==0.5.4 "
      ],
      "metadata": {
        "id": "CGO4qe-DH6Pv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install pyopenjtalk==0.2\n",
        "!pip install pypinyin==0.44.0 \n",
        "!pip install parallel_wavegan==0.5.4 \n",
        "!pip install gdown==4.4.0"
      ],
      "metadata": {
        "id": "oRmBj-K4HVlj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install espnet_model_zoo"
      ],
      "metadata": {
        "id": "2rDPl0MyvSmv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BYLn3bL-qQjN"
      },
      "source": [
        "## Single speaker TTS model demo\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "as4iFXid0m4f"
      },
      "source": [
        "### TTS Model\n",
        "\n",
        "You can try end-to-end text2wav model & combination of text2mel and vocoder.  \n",
        "If you use text2wav model, you do not need to use vocoder (automatically disabled).\n",
        "\n",
        "**Text2wav models**:\n",
        "- VITS\n",
        "\n",
        "**Text2mel models**:\n",
        "- Tacotron2\n",
        "- Transformer-TTS\n",
        "- (Conformer) FastSpeech\n",
        "- (Conformer) FastSpeech2\n",
        "\n",
        "**Vocoders**:\n",
        "- Griffin Lim\n",
        "- Parallel WaveGAN\n",
        "- Multi-band MelGAN\n",
        "- HiFiGAN\n",
        "- Style MelGAN.\n",
        "\n",
        "In this demo, we will only experiment with the English TTS model, but ESPnet-TTS supports multiple languages like Japanese and Mandarin.\n",
        "\n",
        "> The terms of use follow that of each corpus. ESPnet-TTS use the following corpora:\n",
        "- `ljspeech_*`: LJSpeech dataset \n",
        "  - https://keithito.com/LJ-Speech-Dataset/\n",
        "- `jsut_*`: JSUT corpus\n",
        "  - https://sites.google.com/site/shinnosuketakamichi/publication/jsut\n",
        "- `jvs_*`: JVS corpus + JSUT corpus\n",
        "  - https://sites.google.com/site/shinnosuketakamichi/research-topics/jvs_corpus\n",
        "  - https://sites.google.com/site/shinnosuketakamichi/publication/jsut\n",
        "- `tsukuyomi_*`: つくよみちゃんコーパス + JSUT corpus\n",
        "  - https://tyc.rei-yumesaki.net/material/corpus/\n",
        "  - https://sites.google.com/site/shinnosuketakamichi/publication/jsut\n",
        "- `csmsc_*`: Chinese Standard Mandarin Speech Corpus\n",
        "  - https://www.data-baker.com/open_source.html \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J-Bvca5mE7bT"
      },
      "source": [
        "#@title Download English model { run: \"auto\" }\n",
        "lang = 'English'\n",
        "tag = \"kan-bayashi/ljspeech_vits\" #@param [\"kan-bayashi/ljspeech_tacotron2\", \"kan-bayashi/ljspeech_fastspeech\", \"kan-bayashi/ljspeech_vits\"]\n",
        "vocoder_tag = \"none\" #@param [\"none\", \"parallel_wavegan/ljspeech_parallel_wavegan.v1\"]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!gdown --id \"1PjT9FX13d7Mv6loCs-wv5R_v3QrmLixf&confirm=t\" -O /content/tts_model.zip\n",
        "!unzip /content/tts_model.zip -d /content/tts_model"
      ],
      "metadata": {
        "id": "YLJTnc8UUudD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g9S-SFPe0z0w"
      },
      "source": [
        "### Model Setup"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z64fD2UgjJ6Q"
      },
      "source": [
        "from espnet2.bin.tts_inference import Text2Speech\n",
        "from espnet2.utils.types import str_or_none\n",
        "\n",
        "text2speech = Text2Speech.from_pretrained(\n",
        "    train_config=\"/content/tts_model/exp/tts_train_vits_raw_phn_tacotron_g2p_en_no_space/config.yaml\",\n",
        "    model_file=\"/content/tts_model/exp/tts_train_vits_raw_phn_tacotron_g2p_en_no_space/train.total_count.ave_10best.pth\",\n",
        "    vocoder_tag=str_or_none(vocoder_tag),\n",
        "    device=\"cuda\",\n",
        "    # Only for Tacotron 2 & Transformer\n",
        "    threshold=0.5,\n",
        "    # Only for Tacotron 2\n",
        "    minlenratio=0.0,\n",
        "    maxlenratio=10.0,\n",
        "    use_att_constraint=False,\n",
        "    backward_window=1,\n",
        "    forward_window=3,\n",
        "    # Only for FastSpeech & FastSpeech2 & VITS\n",
        "    speed_control_alpha=1.0,\n",
        "    # Only for VITS\n",
        "    noise_scale=0.333,\n",
        "    noise_scale_dur=0.333,\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FMaT0Zev021a"
      },
      "source": [
        "### Synthesis  (✅ Checkpoint 1 (2 point))\n",
        "\n",
        "Run inference of pretrained single-speaker TTS model. Please experiment with running TTS model on different utterances. Provide some examples of failure cases and plot spectrogram and waveform of the utterances for both successful and failure cases. (1 point)\n",
        "\n",
        "Please also discuss possible explanation of these failure cases. (1 point)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vrRM57hhgtHy"
      },
      "source": [
        "import time\n",
        "import torch\n",
        "\n",
        "# decide the input sentence by yourself\n",
        "print(f\"Input your favorite sentence in {lang}.\")\n",
        "x = input()\n",
        "\n",
        "# synthesis\n",
        "with torch.no_grad():\n",
        "    start = time.time()\n",
        "    wav = text2speech(x)[\"wav\"]\n",
        "rtf = (time.time() - start) / (len(wav) / text2speech.fs)\n",
        "print(f\"RTF = {rtf:5f}\")\n",
        "\n",
        "# let us listen to generated samples\n",
        "from IPython.display import display, Audio\n",
        "display(Audio(wav.view(-1).cpu().numpy(), rate=text2speech.fs))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### TTS Model selection"
      ],
      "metadata": {
        "id": "kcoyYzuij6HE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Question2  (✅ Checkpoint 2 (1 point))\n",
        "\n",
        "Please experiment with running different TTS models like Tacotron or FastSpeech. Please also experiment both with Griffin Lim and Parallel WaveGAN vocoder. Please discuss which is better and why."
      ],
      "metadata": {
        "id": "qPpe9cTGjY8R"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YJA6Ii88kKMR"
      },
      "source": [
        "#@title Download English model { run: \"auto\" }\n",
        "lang = 'English'\n",
        "tag = \"kan-bayashi/ljspeech_tacotron2\" #@param [\"kan-bayashi/ljspeech_tacotron2\", \"kan-bayashi/ljspeech_fastspeech\", \"kan-bayashi/ljspeech_vits\"]\n",
        "vocoder_tag = \"none\" #@param [\"none\", \"parallel_wavegan/ljspeech_parallel_wavegan.v1\"]\n",
        "# when vocoder_tag is none, Griffin Lim algorithm is used"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!gdown --id \"1PXsSaulipN31HnQ8YWwsi9Ndb3B2My-J&confirm=t\" -O /content/tts_tacotron_model.zip\n",
        "!unzip /content/tts_tacotron_model.zip -d /content/tts_tacotron_model\n",
        "#For fastspeech model run the commented lines below\n",
        "#!gdown --id \"13Jek_NbI8Qai42v4GKYxx3-jXOun5m2-&confirm=t\" -O /content/tts_fastspeech_model.zip\n",
        "#!unzip /content/tts_fastspeech_model.zip -d /content/tts_fastspeech_model"
      ],
      "metadata": {
        "id": "ltWGfzNIlbY1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from espnet2.bin.tts_inference import Text2Speech\n",
        "from espnet2.utils.types import str_or_none\n",
        "!ln -sf /content/tts_tacotron_model/exp .\n",
        "text2speech = Text2Speech.from_pretrained(\n",
        "    # model_tag=str_or_none(tag),\n",
        "    train_config=\"/content/tts_tacotron_model/exp/tts_train_tacotron2_raw_phn_tacotron_g2p_en_no_space/config.yaml\",\n",
        "    model_file=\"/content/tts_tacotron_model/exp/tts_train_tacotron2_raw_phn_tacotron_g2p_en_no_space/199epoch.pth\",\n",
        "    vocoder_tag=str_or_none(vocoder_tag),\n",
        "    device=\"cuda\",\n",
        "    # Only for Tacotron 2 & Transformer\n",
        "    threshold=0.5,\n",
        "    # Only for Tacotron 2\n",
        "    minlenratio=0.0,\n",
        "    maxlenratio=10.0,\n",
        "    use_att_constraint=False,\n",
        "    backward_window=1,\n",
        "    forward_window=3,\n",
        "    # Only for FastSpeech & FastSpeech2 & VITS\n",
        "    speed_control_alpha=1.0,\n",
        "    # Only for VITS\n",
        "    noise_scale=0.333,\n",
        "    noise_scale_dur=0.333,\n",
        ")\n",
        "# For fastspeech model run the commented lines below\n",
        "# from espnet2.bin.tts_inference import Text2Speech\n",
        "# from espnet2.utils.types import str_or_none\n",
        "# !ln -sf /content/tts_fastspeech_model/exp .\n",
        "# text2speech = Text2Speech.from_pretrained(\n",
        "#     # model_tag=str_or_none(tag),\n",
        "#     train_config=\"/content/tts_fastspeech_model/exp/tts_train_fastspeech_raw_phn_tacotron_g2p_en_no_space/config.yaml\",\n",
        "#     model_file=\"/content/tts_fastspeech_model/exp/tts_train_fastspeech_raw_phn_tacotron_g2p_en_no_space/1000epoch.pth\",\n",
        "#     vocoder_tag=str_or_none(vocoder_tag),\n",
        "#     device=\"cuda\",\n",
        "#     # Only for Tacotron 2 & Transformer\n",
        "#     threshold=0.5,\n",
        "#     # Only for Tacotron 2\n",
        "#     minlenratio=0.0,\n",
        "#     maxlenratio=10.0,\n",
        "#     use_att_constraint=False,\n",
        "#     backward_window=1,\n",
        "#     forward_window=3,\n",
        "#     # Only for FastSpeech & FastSpeech2 & VITS\n",
        "#     speed_control_alpha=1.0,\n",
        "#     # Only for VITS\n",
        "#     noise_scale=0.333,\n",
        "#     noise_scale_dur=0.333,\n",
        "# )"
      ],
      "metadata": {
        "id": "Vbn3tiE7kV0v"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import time\n",
        "import torch\n",
        "\n",
        "# decide the input sentence by yourself\n",
        "print(f\"Input your favorite sentence in {lang}.\")\n",
        "x = input()\n",
        "\n",
        "# synthesis\n",
        "with torch.no_grad():\n",
        "    start = time.time()\n",
        "    wav = text2speech(x)[\"wav\"]\n",
        "rtf = (time.time() - start) / (len(wav) / text2speech.fs)\n",
        "print(f\"RTF = {rtf:5f}\")\n",
        "\n",
        "# let us listen to generated samples\n",
        "from IPython.display import display, Audio\n",
        "display(Audio(wav.view(-1).cpu().numpy(), rate=text2speech.fs))"
      ],
      "metadata": {
        "id": "tOvr154SnO0l"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3TTAygALqY6T"
      },
      "source": [
        "## Multi-speaker Model Demo"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rSEZYh22n4gn"
      },
      "source": [
        "### Model Selection\n",
        "\n",
        "Now we provide only English multi-speaker pretrained model.\n",
        "\n",
        "> The terms of use follow that of each corpus. We use the following corpora:\n",
        "- `libritts_*`: LibriTTS corpus\n",
        "  - http://www.openslr.org/60\n",
        "- `vctk_*`: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit\n",
        "  - http://www.udialogue.org/download/cstr-vctk-corpus.html\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NSKLmDN9E7bW"
      },
      "source": [
        "#@title English multi-speaker pretrained model { run: \"auto\" }\n",
        "lang = 'English'\n",
        "tag = 'kan-bayashi/vctk_full_band_multi_spk_vits' #@param [\"kan-bayashi/vctk_gst_tacotron2\", \"kan-bayashi/vctk_gst_transformer\", \"kan-bayashi/vctk_xvector_tacotron2\", \"kan-bayashi/vctk_xvector_transformer\", \"kan-bayashi/vctk_xvector_conformer_fastspeech2\", \"kan-bayashi/vctk_gst+xvector_tacotron2\", \"kan-bayashi/vctk_gst+xvector_transformer\", \"kan-bayashi/vctk_gst+xvector_conformer_fastspeech2\", \"kan-bayashi/vctk_multi_spk_vits\", \"kan-bayashi/vctk_full_band_multi_spk_vits\", \"kan-bayashi/libritts_xvector_transformer\", \"kan-bayashi/libritts_xvector_conformer_fastspeech2\", \"kan-bayashi/libritts_gst+xvector_transformer\", \"kan-bayashi/libritts_gst+xvector_conformer_fastspeech2\", \"kan-bayashi/libritts_xvector_vits\"] {type:\"string\"}\n",
        "vocoder_tag = \"none\" #@param [\"none\", \"parallel_wavegan/vctk_parallel_wavegan.v1.long\", \"parallel_wavegan/vctk_multi_band_melgan.v2\", \"parallel_wavegan/vctk_style_melgan.v1\", \"parallel_wavegan/vctk_hifigan.v1\", \"parallel_wavegan/libritts_parallel_wavegan.v1.long\", \"parallel_wavegan/libritts_multi_band_melgan.v2\", \"parallel_wavegan/libritts_hifigan.v1\", \"parallel_wavegan/libritts_style_melgan.v1\"] {type:\"string\"}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!gdown --id \"1fzyyjLvrT_jldw4lfOD1P8FK2MGoIZO_&confirm=t\" -O /content/tts_multi-speaker_model.zip\n",
        "!unzip /content/tts_multi-speaker_model.zip -d /content/tts_multi-speaker_model"
      ],
      "metadata": {
        "id": "6lCLJ9o6Ykpf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GcshmgYpoVzh"
      },
      "source": [
        "### Model Setup"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kfJFD4QroNhJ"
      },
      "source": [
        "from espnet2.bin.tts_inference import Text2Speech\n",
        "from espnet2.utils.types import str_or_none\n",
        "\n",
        "text2speech = Text2Speech.from_pretrained(\n",
        "    train_config=\"/content/tts_multi-speaker_model/exp/tts_train_full_band_multi_spk_vits_raw_phn_tacotron_g2p_en_no_space/config.yaml\",\n",
        "    model_file=\"/content/tts_multi-speaker_model/exp/tts_train_full_band_multi_spk_vits_raw_phn_tacotron_g2p_en_no_space/train.total_count.ave_10best.pth\",\n",
        "    vocoder_tag=str_or_none(vocoder_tag),\n",
        "    device=\"cuda\",\n",
        "    # Only for Tacotron 2 & Transformer\n",
        "    threshold=0.5,\n",
        "    # Only for Tacotron 2\n",
        "    minlenratio=0.0,\n",
        "    maxlenratio=10.0,\n",
        "    use_att_constraint=False,\n",
        "    backward_window=1,\n",
        "    forward_window=3,\n",
        "    # Only for FastSpeech & FastSpeech2 & VITS\n",
        "    speed_control_alpha=1.0,\n",
        "    # Only for VITS\n",
        "    noise_scale=0.333,\n",
        "    noise_scale_dur=0.333,\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gdaMNwrtuZhY"
      },
      "source": [
        "### Speaker selection\n",
        "\n",
        "For multi-speaker model, we need to provide X-vector and/or the reference speech to decide the speaker characteristics.  \n",
        "For X-vector, you can select the speaker from the dumped x-vectors.  \n",
        "For the reference speech, you can use any speech but please make sure the sampling rate is matched."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZzoAd1rgObcP"
      },
      "source": [
        "import glob\n",
        "import os\n",
        "import numpy as np\n",
        "import kaldiio\n",
        "\n",
        "# Get model directory path\n",
        "from espnet_model_zoo.downloader import ModelDownloader\n",
        "d = ModelDownloader()\n",
        "# model_dir = os.path.dirname(d.download_and_unpack(tag)[\"train_config\"])\n",
        "\n",
        "# X-vector selection\n",
        "spembs = None\n",
        "if text2speech.use_spembs:\n",
        "    xvector_ark = [p for p in glob.glob(f\"/content/tts_multi-speaker_model/dump/**/spk_xvector.ark\", recursive=True) if \"tr\" in p][0]\n",
        "    xvectors = {k: v for k, v in kaldiio.load_ark(xvector_ark)}\n",
        "    spks = list(xvectors.keys())\n",
        "\n",
        "    # randomly select speaker\n",
        "    random_spk_idx = np.random.randint(0, len(spks))\n",
        "    spk = spks[random_spk_idx]\n",
        "    spembs = xvectors[spk]\n",
        "    print(f\"selected spk: {spk}\")\n",
        "\n",
        "# Speaker ID selection\n",
        "sids = None\n",
        "if text2speech.use_sids:\n",
        "    spk2sid = glob.glob(f\"/content/tts_multi-speaker_model/dump/**/spk2sid\", recursive=True)[0]\n",
        "    with open(spk2sid) as f:\n",
        "        lines = [line.strip() for line in f.readlines()]\n",
        "    sid2spk = {int(line.split()[1]): line.split()[0] for line in lines}\n",
        "    \n",
        "    # randomly select speaker\n",
        "    sids = np.array(np.random.randint(1, len(sid2spk)))\n",
        "    spk = sid2spk[int(sids)]\n",
        "    print(f\"selected spk: {spk}\")\n",
        "\n",
        "# Reference speech selection for GST\n",
        "speech = None\n",
        "if text2speech.use_speech:\n",
        "    # you can change here to load your own reference speech\n",
        "    # e.g.\n",
        "    # import soundfile as sf\n",
        "    # speech, fs = sf.read(\"/path/to/reference.wav\")\n",
        "    # speech = torch.from_numpy(speech).float()\n",
        "    speech = torch.randn(50000,) * 0.01"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W6G-1YW9ocYV"
      },
      "source": [
        "### Synthesis(✅ Checkpoint3 (2 point))\n",
        "\n",
        "Run inference of pretrained multi-speaker TTS model on more than one speaker id. Plot spectrogram and waveform of the synthesized speech for these speaker ids."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o87zK1NLobne"
      },
      "source": [
        "import time\n",
        "import torch\n",
        "\n",
        "# decide the input sentence by yourself\n",
        "print(f\"Input your favorite sentence in {lang}.\")\n",
        "x = input()\n",
        "\n",
        "# synthesis\n",
        "with torch.no_grad():\n",
        "    start = time.time()\n",
        "    wav = text2speech(x, speech=speech, spembs=spembs, sids=sids)[\"wav\"]\n",
        "rtf = (time.time() - start) / (len(wav) / text2speech.fs)\n",
        "print(f\"RTF = {rtf:5f}\")\n",
        "\n",
        "# let us listen to generated samples\n",
        "from IPython.display import display, Audio\n",
        "display(Audio(wav.view(-1).cpu().numpy(), rate=text2speech.fs))"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}