Skip to content

Latest commit

 

History

History
800 lines (584 loc) · 33.4 KB

readme.md

File metadata and controls

800 lines (584 loc) · 33.4 KB
Raw

VoxNovel

Voxnovel_Logo

📋 Overview

VoxNovel is an innovative program that leverages the capabilities of booknlp to analyze literature, attribute quotations to specific characters, and generate a tailored audiobook where each character has a distinct voice through coqui tts. This not only provides an immersive audiobook experience but also brings each character to life with a unique voice, making the listening experience much more engaging.

🗣️ Included TTS Models

All Coqui TTS models-(Tacotron, Tacotron2, Glow-TTS, Speedy-Speech, Align-TTS, FastPitch, FastSpeech, FastSpeech2, SC-GlowTTS, Capacitron, OverFlow, Neural HMM TTS, Delightful TTS, ⓍTTS, VITS, 🐸 YourTTS, 🐢 Tortoise, 🐶 Bark), and STYLETTS2.

🌍🎙️ Accents you can give each character with the default cloning model (XTTS) - They also allow them to speak these languages, but the quotation attribution won't correctly identify for anything thats not English. English (en), Spanish (es), French (fr), German (de), Italian (it), Portuguese (pt), Polish (pl), Turkish (tr), Russian (ru), Dutch (nl), Czech (cs), Arabic (ar), Chinese (zh-cn), Japanese (ja), Hungarian (hu), Korean (ko)

outputs as a m4b with all book metadata and chapters, example output file in a audiobook player app

Example_of_output_in_audiobook_program

(as well as a folder of individual mp4 chatper files with ebook image embedded in them if you want that)

🔊 DEMOS

High Quality XTTS V2 Demos

Guardians_of_Ga.Hoole_10._Lasky._Kathryn_-_Coming_Of_Hoole_chapter_4.mp4
🔊🎉 More Demo Audio files :)

High Quality Tortoise Demos

272463996-c4f8dfdf-c5bd-4771-ab1a-6131c22a67b2.mp4

Super fast audio Balacoon Demos

271878548-53b694a9-5ddd-4174-82bc-07aff22d2330.mp4
271876316-530e8781-c77c-4424-89cd-a02df363b0bf.mp4

**Super High Quality testing with fine tuned models **

AUDIO_5811.mp4

You can fine-tune your own Xtts models with around 6+ minutes of audio for free with this colab https://colab.research.google.com/drive/1GiI4_X724M8q2W-zZ-jXo7cWTV7RfaH-

Edit: that colab doesn't work anymore: use my version that provides a fix: https://colab.research.google.com/drive/1sqQqzupo2pdjgggkrbM60sU6sBFYo3su?usp=sharing

🤖 Headless VoxNovel Google Colab

Explore and run the interactive version of the Headless VoxNovel project directly on Google Colab! Get started here.

GUI

gui_1_select_file

GUI Part 1 (BookNLP Processor) Info/Features -"Process File" button: Click and it'll ask you to select a ebook file.

gui_2_finetune

Manual Speaker Assignment Correction Tool (GUI 1.5)

This GUI is for the manual correction of speaker assignments if quotes are incorrectly assigned by BookNLP. It reads the book.csv file containing the books extracted quotes and speaker information, and allows you to visually inspect and modify speaker assignments as needed before being passed to the next TTS step.

Key Features:

  • Scrollable Text Display: Allows users to view the book's text with color-coded speaker assignments.
  • Speaker Selection: Users can select a new speaker from a dropdown menu to reassign specific lines.
  • Checkable Quotes: Lines from the book are displayed with checkboxes, enabling the selection of multiple lines for speaker reassignment.
  • Speaker Color Coding: Each speaker is assigned a unique color for easy identification.
  • Buttons for Action:
    • Update Selected Speakers: Apply the selected speaker to all checked lines.
    • Deselect All: Uncheck all selected lines.
    • Continue: Save changes and exit the tool.

How to Use:

  1. Select Lines: Check the boxes next to the lines you want to change.
  2. Choose Speaker: Select the desired speaker from the dropdown menu.
  3. Apply Changes: Click "Update Selected Speakers" to apply the changes.
  4. Review: The text will update to reflect the changes.
  5. Deselect: Click "Deselect All" to clear your selections.
  6. Finish: Once satisfied with the corrections, click "Continue" to save and exit.

gui_2_finetune

GUI Part 2 (Coqui TTS GUI) Info/Features
  • Select TTS Model Dropdown: This selects the TTS model that will be used for voice cloning.
  • Include fast Voice Models Checkbox: (Fast generate at cost of audio quality) Click this to be able to see every other model and singular voices supported by Coqui TTS.
    • It will update the "Select TTS Model" Dropdown for voice cloning models to also include (List of values to be added).
    • It will update the Dropdown for voices to select for each character to also include (List of values to be added).
  • Make all audio generate with Narrator voice Checkbox: This will make every character's audio be generated with the voice you have selected for the Narrator when you click the "Generate audio" button.
  • Clone new voice Button: Click this to add a new voice you can clone (make sure you have a reference audio file on hand).
  • Add Fine-tuned Xtts model to voice actor Button: If you have a folder containing all the parameters of a fine-tuned Xtts model of a specific voice, then you can click this to make that voice actor clone with that fine-tuned Xtts model, to provide much better voice cloning results.
  • Character voices Dropdowns: These are the dropdowns for selecting the Voice Actor (and the Accent of each character if using XTTS).
    • (1): The Voice actors available to select from for this character. (Default value is audio selected based on inferred gender of character being: "F, M, Other").
      • When you select a voice It will play the audio sample of that voice, if it's a fast voice model voice and a refrence audio does not exist, then it will generate one to play.
    • (2): The Accents available to select from for this character. (Optional, Default is English).
  • Chapter Delimiter Field: Will change the default chapter delimiter (The string that's used to identify chapters).
  • Silence Duration in milliseconds (ms) Field: This will change the amount of milliseconds in between each combined chunk of audio.
  • Select TTS Language Dropdown: This will let you select the default Accent used for every character which has not had the Accent manually selected for.
  • Loading bar: Will give an approximate amount of time left. (Estimate, you probably won't see accurate predictions until it's been running for 5 min).
  • Annotated book preview Block: This will show the entirety of the book with each character's lines color-coded.
    • You can click on a line while the audiobook is being generated to hear what that generated line sounds like. But only if the line has already had audio generated for it; if not, it'll play nothing.
  • Load Book Button: Clicking this will reload the color-coded annotated book view, it will just randomize the selected colors for each character's lines.
  • Generate Audio Button: Will start generating the full audiobook.
  • Select random voices Button (Will only be visible if the "include fast Voice Models" checkbox is checked): Will Select an auto-gender-inferred fast model voice for every character except for the narrator's voice.

gui_3_run

GUI Part 3 (Book Viewer) Info/Features -It's hard to explain its more of a playground if you mess around with it then you should get how it works. But it can be used to fine tune the audiobook -Close out of the window when your done with it.

📦 SetUp Install

🤖 Headless VoxNovel Google Colab

Explore and run the interactive version of the Headless VoxNovel project directly on Google Colab! Get started here.

🐳 Docker (Sound not working in gui yet)
🐳 Headless Docker
Docker headless m1 🍏Mac

  1. cd ~

  2. git clone https://github.com/DrewThomasson/VoxNovel.git

  3. sudo docker run -v "$HOME/VoxNovel:/VoxNovel/" -it athomasson2/voxnovel:headless_m1_v2

Headless Docker 🐧 Linux/Intel 🍏Mac

For Headless Docker on only cpu

  1. cd ~

  2. git clone https://github.com/DrewThomasson/VoxNovel.git

  3. sudo docker run -v "$HOME/VoxNovel:/VoxNovel/" -it athomasson2/voxnovel:latest_headless

For headless docker with gpu speedup if you have a nvida gpu

  1. cd ~

  2. git clone https://github.com/DrewThomasson/VoxNovel.git

  3. sudo docker run --gpus all -v "$HOME/VoxNovel:/VoxNovel/" -it athomasson2/voxnovel:latest_headless

Headless Docker 🖥️ Windows

Installation and Setup on Windows (PowerShell)

Follow these steps to set up the VoxNovel project on a Windows system using PowerShell:

  1. Navigate to your user profile directory:

    cd $env:USERPROFILE

  2. Clone the VoxNovel repository from GitHub:

    git clone https://github.com/DrewThomasson/VoxNovel.git

Running VoxNovel in Docker

For Headless Operation on CPU

To run the VoxNovel application in a Docker container on your CPU:

docker run -v "${env:USERPROFILE}/VoxNovel/:/VoxNovel/" -it athomasson2/voxnovel:latest_headless

For Headless Operation with NVIDIA GPU Speedup

If you have an NVIDIA GPU and want to accelerate processing, use the following command:

docker run --gpus all -v "${env:USERPROFILE}/VoxNovel/:/VoxNovel/" -it athomasson2/voxnovel:latest_headless
🐳 GUI Docker (Sound not working in gui yet)
🐧 Linux Docker 1. `cd ~`
  1. git clone https://github.com/DrewThomasson/VoxNovel.git
  2. sudo docker run --gpus all -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix -v /dev/snd:/dev/snd --device /dev/snd -v "$HOME/VoxNovel:/VoxNovel/" -it athomasson2/voxnovel:latest
🍏 Mac Docker

Setting Up GUI Applications with Docker on macOS

This guide provides instructions on how to run a Docker container with a graphical user interface on macOS using XQuartz for X11 forwarding and volume mounting.

Install XQuartz

  1. Download and install XQuartz from XQuartz website.
  2. Open XQuartz.
  3. Go to XQuartz -> Preferences.
  4. In the Security tab, enable Allow connections from network clients.
  5. Restart XQuartz to apply these settings.

Configure and Run the Docker Container

Allow Docker to Connect to XQuartz

Open a terminal and run the following command to allow connections from your local machine to XQuartz:

xhost + $(ifconfig en0 | grep inet | awk '$1=="inet" {print $2}')

Start the Docker Container

Run the following command to start your Docker container. This command configures the GUI to display on your host and mounts the necessary directories:

cd ~
git clone https://github.com/DrewThomasson/VoxNovel.git
docker run -e DISPLAY=$(ifconfig en0 | grep inet | awk '$1=="inet" {print $2}'):0 \
           -v /tmp/.X11-unix:/tmp/.X11-unix \
           -v "/Users/$(whoami)/VoxNovel:/VoxNovel" \
           athomasson2/voxnovel:latest

Notes

  • XQuartz Configuration: Ensure that XQuartz is configured to allow network clients before attempting to connect.
  • Directory Existence: Verify that the directory /Users/$(whoami)/VoxNovel exists on your Mac. If not, create it or adjust the volume mount path in the Docker command as needed.
  • Firewall and Security: If you face connectivity issues, check any firewall settings and security preferences that might block the connections.
🪟 Windows Docker

  1. Install VcXsrv:

    choco install vcxsrv
    • First install VcXsrv and configure it to allow connections.
    How to setup VcXsrv

    After installing VcXsrv, it typically launches automatically. You can confirm it's running by checking for its icon in the system tray, usually located near the clock in the taskbar. It may also start automatically when you log in to your system.

    To ensure it's configured to allow connections from Docker containers, follow these steps:

    1. Right-click on the VcXsrv icon in the system tray.
    2. Select "XLaunch" to open the configuration wizard.
    3. In the configuration wizard, select "Multiple windows" and proceed to the next step.
    4. Choose your preferred settings for display number and screen.
    5. In the "Extra settings" window, make sure to check the box labeled "Disable access control" to allow connections from Docker containers.
    6. Complete the configuration by clicking "Finish" and then "Save configuration" when prompted.

    With these settings, VcXsrv should be running and configured to allow connections from Docker containers. You can now proceed with running your Docker commands requiring GUI support.

  2. Change to your home directory:

    cd $HOME

  3. Clone the repository:

    git clone https://github.com/DrewThomasson/VoxNovel.git

  4. Run the Docker container:

    docker run -e DISPLAY=host.docker.internal:0 -v "/Users/$(whoami)/VoxNovel:/VoxNovel/" -it athomasson2/voxnovel:latest
🐧 Linux

💻 Single Command Ubuntu Install

(Do not use if you already have Miniconda installed.)

To install VoxNovel on Ubuntu, you can use the following single command:

yes | wget -O - https://raw.githubusercontent.com/DrewThomasson/VoxNovel/main/shell_install_scripts/Ubuntu-install.sh | bash

Desktop shortcut

-That single install script above should also create a shortcut for the application as well.

Manual Launch Option

OR You can manually launch the application in the terminal with the following command:

cd ~/VoxNovel && conda activate VoxNovel && python gui_run.py

or manual install:

  1. sudo apt-get install calibre
  2. sudo apt-get install ffmpeg
  3. conda create --name VoxNovel python=3.10
  4. conda activate VoxNovel
  5. git clone https://github.com/DrewThomasson/VoxNovel.git
  6. cd VoxNovel
  7. pip install bs4
  8. pip install styletts2
  9. pip install tts==0.21.3
  10. pip install booknlp==1.0.7.1
  11. pip install -r Ubuntu_requirements.txt
  12. python -m spacy download en_core_web_sm
🈳 For non Latin-based Languages TTS support (Optional)

Install Mecab for (Non Latin-based Languages tts support)(Optional):

  • Ubuntu: sudo apt-get install -y mecab libmecab-dev mecab-ipadic-utf8

(For non Latin-based Languages tts support)(Optional)
python -m unidic download

pip install mecab mecab-python3 unidic
🔲 Steam Deck) (x86_64 Arch Linux)
  1. sudo -v && wget -nv -O- https://download.calibre-ebook.com/linux-installer.sh | sudo sh /dev/stdin
  2. also download it from the discovery store or flatpac I did both on my steam deck
  3. mkdir -p ~/miniconda3
  4. wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
  5. bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
  6. rm -rf ~/miniconda3/miniconda.sh
  7. ~/miniconda3/bin/conda init bash
  8. ~/miniconda3/bin/conda init zsh
  9. restart the terminal(close out and open a new window)
  10. conda create --name VoxNovel python=3.10
  11. conda activate VoxNovel
  12. git clone https://github.com/DrewThomasson/VoxNovel.git
  13. cd VoxNovel
  14. sudo pacman -S espeak-ng(make sure you have pacman fully working, there should be stuff online for the steam deck) guide on getting pacman wokring on steam deck(https://www.reddit.com/r/SteamDeck/comments/t8al0i/install_arch_packages_on_your_steam_deck/) you might have to reset the keys if something goes wrong with that: Resetting all the keys

Remove or reset all the keys installed in your system by removing the /etc/pacman.d/gnupg directory (as root) and by rerunning pacman-key --init followed by pacman-key --populate to re-add the default keys. 15. pip install styletts2 16. pip install tts==0.21.3 17. pip install booknlp==1.0.7.1 18. pip install -r SteamDeck_requirements.txt 19. pip3 install spacy 20. python3 -m spacy download en_core_web_sm 21. sudo pacman -S calibre 22. mv ~/miniconda3/envs/VoxNovel/lib/libstdc++.so.6 ~/miniconda3/envs/tts/lib/libstdc++.so.6.bak

  1. pip install mechanize
  2. pip install bs4
  3. pip install css_parser
🍏 Intel mac

💻 Install on Intel Mac:

Download Intel VoxNovel Installer

Or run the following command in your terminal:

bash <(curl -s https://raw.githubusercontent.com/DrewThomasson/VoxNovel/main/shell_install_scripts/Intel_Mac_Install_VoxNovel.sh)

Once it's complete you should have a desktop shortcut for VoxNovel.

🧹 Uninstall on Intel Mac:

To uninstall, run the following command in your terminal:

bash <(curl -s https://raw.githubusercontent.com/DrewThomasson/VoxNovel/main/shell_install_scripts/uninstall_VoxNovel_Mac.sh)
(🚫 Depricated do not use 🚫) Intel Mac Manual-install Run in this order:
  1. brew install calibre
  2. brew install ffmpeg
  3. conda create --name VoxNovel python=3.10
  4. conda activate VoxNovel
  5. git clone https://github.com/DrewThomasson/VoxNovel.git
  6. cd VoxNovel
  7. pip install styletts2
  8. pip install tts==0.21.3
  9. pip install booknlp==1.0.7.1 9.pip install -r MAC-requirements.txt
  10. pip install spacy 11.python -m spacy download en_core_web_sm
🈳 For non Latin-based Languages TTS support (Optional)

Install Mecab for (Non Latin-based Languages tts support)(Optional):

  • macOS: brew install mecab, brew install mecab-ipadic

(For non Latin-based Languages tts support)(Optional)
python -m unidic download

pip install mecab mecab-python3 unidic
🍏 Apple Silicon Mac (Tested on 2020 M1 pro 8gb ram)

💻 Install on Apple Silicon Mac:

Download Apple Silicon VoxNovel Installer

Or run the following command in your terminal:

bash <(curl -s https://raw.githubusercontent.com/DrewThomasson/VoxNovel/main/shell_install_scripts/Apple_silicone_VoxNovel_install.sh)

Once it's complete you should have a desktop shortcut for VoxNovel.

🧹 Uninstall on Apple Silicon Mac:

To uninstall, run the following command in your terminal:

bash <(curl -s https://raw.githubusercontent.com/DrewThomasson/VoxNovel/main/shell_install_scripts/uninstall_VoxNovel_Mac.sh)
(🚫 Depricated do not use 🚫) Apple Silicon Manual-install

Run in this order:

  1. brew install calibre(You might have to also install it manually from their site if this doesn't work)
  2. brew install ffmpeg
  3. conda create --name VoxNovel python=3.10
  4. conda activate VoxNovel
  5. git clone https://github.com/DrewThomasson/VoxNovel.git
  6. cd VoxNovel
  7. pip install tensorflow-macos (Also optional pip install tensorflow-metal But so far I haven't gotten gpu speedup yet
  8. pip install styletts2
  9. pip install tts==0.21.3
  10. pip install --no-dependencies booknlp==1.0.7.1
  11. pip install transformers==4.30.0
  12. pip install tensorflow
  13. pip install -r MAC-requirements.txt
  14. pip install ebooklib bs4 epub2txt pygame moviepy spacy
  15. python -m spacy download en_core_web_sm
🈳 For non Latin-based Languages TTS support (Optional)

Install Mecab for (Non Latin-based Languages tts support)(Optional):

  • macOS: brew install mecab, brew install mecab-ipadic (For non Latin-based Languages tts support)(Optional)
    python -m unidic download
pip install mecab mecab-python3 unidic
🪟 Windows 11

Because of BookNLP Windows issues, all of this will be run in WSL (don't worry, it's still easy).

🎥 Watch the installation video here

  1. In your PowerShell, paste:

    wsl --install

    to install WSL. (You might be prompted by your system to enable virtualization in your BIOS if it's available, as it is needed to run WSL on Windows.)

  2. After setting your username and password, open WSL and paste this command for a single command install:

    yes | wget -O - https://raw.githubusercontent.com/DrewThomasson/VoxNovel/main/shell_install_scripts/Ubuntu-install.sh | bash

  3. (Optional only for Nvida graphics cards Do not run this command if you don't have a Nvidia graphics card) Install the NVIDIA CUDA toolkit (required for Nvidia GPU acceleration):

    sudo apt install nvidia-cuda-toolkit

  4. Make sure you are in the VoxNovel conda environment:(If 'conda: command not found ' IE- conda is not seen as a command then try closing out of the current powershell window and relaunching the wsl env with [ wsl -d Ubuntu ]

    conda activate VoxNovel

  5. Navigate to the VoxNovel folder (if not already there):

    cd ~ && cd VoxNovel

  6. Now just run one of the two programs shown below ⬇️

🚀 To Run the program

python gui_run.py

🚀 Or to run headless

python headless_voxnovel.py

🌐 Access WSL Ubuntu Files from Windows

You can access your WSL Ubuntu files directly in Windows File Explorer by entering the following path in the address bar:

\\wsl.localhost\Ubuntu\home\

The output audiobook files will be located under VoxNovel\output_audiobooks in the wsl env

🎯 To Create VoxNovel Windows Desktop Shortcut

Run this command in powershell

Invoke-Expression (Invoke-WebRequest -Uri "https://raw.githubusercontent.com/DrewThomasson/VoxNovel/main/shell_install_scripts/Windows-install-scripts/create_desktop_shortcut.ps1").Content

🗑️ Uninstallation:

To remove everything, run the following command in PowerShell:

wsl --unregister Ubuntu

This will completely remove the Ubuntu environment where the application is stored. 🚮

🛠️ Troubleshooting WSL

If you have trouble with the WSL environment:

  1. List all WSL environments:

    wsl --list --verbose

  2. Remove a specific WSL environment (e.g., Ubuntu):

    wsl --unregister <distro_name>

  3. Reinstall WSL:

    wsl --install

To launch WSL anytime you need to run this program, you can use the search bar in Windows to find and launch "WSL" or run:

wsl
🈳 For non Latin-based Languages TTS support (Optional)

Install Mecab for (Non Latin-based Languages tts support)(Optional):

  • sudo apt-get install -y mecab libmecab-dev mecab-ipadic-utf8

(For non Latin-based Languages tts support)(Optional)
python -m unidic download

pip install mecab mecab-python3 unidic

🚀 To Run the program

python gui_run.py

🚀 Or to run headless

python headless_voxnovel.py
Running with Low VRAM (4 GB)

Modifications

  • Turns out once you set the device it stays like that for the full program.
  • So, I've split the program into two Python programs: one CPU and one GPU. I've tested this on my (4GB VRAM GPU) and this solution works. at least on my end I really hope it works on your end. 🙏

To run the fix I've made tailor made for a low Vram GPU situation:

To run the provided scripts on your system, follow these steps in order:

  1. Book Processing (CPU Only):

    • Script: 1CPU_Book_processing.py
    • This script handles the task of only processing the book using BookNLP, specifically forcing it to run on the CPU.
    • Run with python 1CPU_Book_processing.py

  2. Audio Generation (GPU Only):

    • Script: 2GPU_Audio_generation.py
    • This script is dedicated to only generating audio with the GPU and should be run after completing the book processing with 1CPU_Book_processing.py.
    • Run with python 2GPU_Audio_generation.py

Performance Results

Upon running a mini test with an epub file using the above setup, the following performance metrics were observed:

Performance Results

Testing on done with the mini epub file located in the Example_working_files.zip

Task Configuration Time (Seconds)
Book Processing GPU only (GeForce GTX 980), 4GB VRAM, 32GB RAM, Intel i7-8700K 2.922
Audio Generation GPU only (GeForce GTX 980), 4GB VRAM, 32GB RAM, Intel i7-8700K 128.48
Book Processing CPU only, 32GB RAM, Intel i7-8700K 4.964
Audio Generation CPU only, 32GB RAM, Intel i7-8700K 391.4227
To Run the auto program This means all you do is select the book and all the voices will be auto assigned and generated for you.

python auto_noGui_run.py

🌐 Access generated audiobook files

You can access your generated audiobook files in the VoxNovel folder at the location

VoxNovel/output_audiobooks

📚 Supported ebook File Types:

.epub, .pdf, .mobi, .txt, .html, .rtf, .chm, .lit, .pdb, .fb2, .odt, .cbr, .cbz, .prc, .lrf, .pml, .snb, .cbc, .rb, and .tcr,

  • (Best results are from using epub or mobi for auto chapter detection)

Folders

📂 Folders used by the program

/Final_combined_output_audio: This is where all of your chapter audio files will be put in order of chapter num

/output_audiobooks: This is where all of your m4b audiobook files will be stored

/Working_files: Holds all of the working files used by the program while activly running.

  • /Working_files/temp_ebook: Holds all of the individual extracted chapter txt files from the ebook.

/tortoise: Holds all the sample voice files

GUI functions

GUI Part 1 (BookNLP Processor) -"Process File" button: Click and it'll ask you to select a ebook file.
GUI Part 2 (Coqui TTS GUI)
  • Select TTS Model Dropdown: This selects the TTS model that will be used for voice cloning.
  • Include fast Voice Models Checkbox: (Fast generate at cost of audio quality) Click this to be able to see every other model and singular voices supported by Coqui TTS.
    • It will update the "Select TTS Model" Dropdown for voice cloning models to also include (List of values to be added).
    • It will update the Dropdown for voices to select for each character to also include (List of values to be added).
  • Make all audio generate with Narrator voice Checkbox: This will make every character's audio be generated with the voice you have selected for the Narrator when you click the "Generate audio" button.
  • Clone new voice Button: Click this to add a new voice you can clone (make sure you have a reference audio file on hand).
  • Add Fine-tuned Xtts model to voice actor Button: If you have a folder containing all the parameters of a fine-tuned Xtts model of a specific voice, then you can click this to make that voice actor clone with that fine-tuned Xtts model, to provide much better voice cloning results.
  • Character voices Dropdowns: These are the dropdowns for selecting the Voice Actor (and the Accent of each character if using XTTS).
    • (1): The Voice actors available to select from for this character. (Default value is audio selected based on inferred gender of character being: "F, M, Other").
      • When you select a voice It will play the audio sample of that voice, if it's a fast voice model voice and a refrence audio does not exist, then it will generate one to play.
    • (2): The Accents available to select from for this character. (Optional, Default is English).
  • Chapter Delimiter Field: Will change the default chapter delimiter (The string that's used to identify chapters).
  • Silence Duration in milliseconds (ms) Field: This will change the amount of milliseconds in between each combined chunk of audio.
  • Select TTS Language Dropdown: This will let you select the default Accent used for every character which has not had the Accent manually selected for.
  • Loading bar: Will give an approximate amount of time left. (Estimate, you probably won't see accurate predictions until it's been running for 5 min).
  • Annotated book preview Block: This will show the entirety of the book with each character's lines color-coded.
    • You can click on a line while the audiobook is being generated to hear what that generated line sounds like. But only if the line has already had audio generated for it; if not, it'll play nothing.
  • Load Book Button: Clicking this will reload the color-coded annotated book view, it will just randomize the selected colors for each character's lines.
  • Generate Audio Button: Will start generating the full audiobook.
  • Select random voices Button (Will only be visible if the "include fast Voice Models" checkbox is checked): Will Select an auto-gender-inferred fast model voice for every character except for the narrator's voice.
GUI Part 3 (Book Viewer) -It's hard to explain its more of a playground if you mess around with it then you should get how it works. But it can be used to fine tune the audiobook -Close out of the window when your done with it.

🌟 Features

  • Free and entirely locally run
  • Supports all ebook file formats by using calibre
  • Can run on CPU or CUDA GPU
  • Autoselects a starting estimated voice by pronouns per character
  • Supports all models in Coqui TTS and all voices in the models
  • Easily create a new voice actor in seconds through voice cloning in GUI
  • Can play audio by clicking on the text in the book viewer in GUI
  • Ability to regenerate specific lines if they came out weird
  • Ability to add custome fine tuned models for specific voice with the click of a button in GUI
  • Outputs a single file output as m4b to include all the metadata(chapters book image ect)
  • Supports STYLETTS2 as a model you can select from for voice cloning for WICKED FAST speed (even on cpu)
  • Includes 26 default voices for cloning
  • Low vram option(Details in readme)
  • Gui Docker image (Graphical interface-no sound in gui yet)
  • Headless version of docker image
  • Headless version of VoxNovel
  • A Google Colab using the headless version
  • Desktop Shortcut for Ubuntu Linux 🐧
  • Desktop Shortcut for Windows 11 🪟
  • Desktop Shortcut for Mac 🍏
  • Ability to change the character for a line if incorrectly attributed by booknlp
  • All Desktop shortcuts automatically check for updates when run

🔜 Incoming Planned Features

  • Make it so that all the included voices and models already have their premade own demo voices
  • Make it so that the demo audio for the cloned voices is not their reference audio but what their voices sound like generated
  • Using whisper transcriptions to cut hallucinations out of generated audio
  • Incorporating local model to generate sound effects when a book discribes a location or sound effect
  • Adding Save file functionaly

🙏 Special thanks to:

-@sidharthrajaram (for his Styletts2 pip install he created, I couldn't of added styletts2 without him. :) ) (https://github.com/sidharthrajaram/StyleTTS2)