Subtitles for generated audio

[arastoof]

Is there any way to generate subtitles for the audio the TTS generates? Like a timestamp for every word or sentence the TTS speaks. Currently I am using OpenAI's whisper model, and that works fine, but I think it would be much faster if there was something already in the AI. I'm really new to this so sorry if this is a basic question

[Sharrnah]

not sure why you need that since to generate Text to Speech, you already have the Text to show as subtitles.

Or do you mean aligning it to the exact word timings?

[arastoof]

Yes I meant to align it to the word timings. I just wrote some python to do this for me idk if anyone else is interested in doing this:

`from nltk import sent_tokenize
from TTS.api import TTS
import wave
import os
import contextlib
import datetime

#Sample text
thetext = """The quick brown fox jumps over the lazy dog. The dog barks loudly! The fox runs away... but the dog chases it! The fox jumps over a fence and escapes. The dog is left behind, panting heavily.

Meanwhile, in a nearby forest, a group of animals are having a party. There's a rabbit, a squirrel, and a raccoon. They're all dancing to some music and having a great time. Suddenly, they hear a loud noise outside. They look out the window and see the fox running by.

The animals are scared and run to hide. The rabbit hides under a bush, the squirrel climbs up a tree, and the raccoon hides in a hole. The fox runs past them and doesn't even notice them.

After a few minutes, the animals come out of hiding and continue their party. They're relieved that the fox didn't find them.

The end?
"""

#Splits the text into a list using nltk
textlist = sent_tokenize(thetext)

#Loads the model
tts = TTS(model_name="tts_models/en/ljspeech/vits--neon", progress_bar=False)

#Creates a temporary wav file for each of the items spoken in textlist
infiles = []

for i in range(len(textlist)):
tts.tts_to_file(text = textlist[i], file_path = f'temp audio\temptts{i}.wav')
infiles.append(f"temp audio\temptts{i}.wav")

#Gets the durations of all of the temporary wav files individually
durations = [0]

for file in infiles:
with contextlib.closing(wave.open(file,'r')) as f:
frames = f.getnframes()
rate = f.getframerate()
duration = frames / float(rate)
durations.append(duration)

#Concatenates all wav files and saves to sounds.wav
outfile = "sounds.wav"

data= []
for infile in infiles:
w = wave.open(infile, 'rb')
data.append( [w.getparams(), w.readframes(w.getnframes())] )
w.close()

output = wave.open(outfile, 'wb')
output.setparams(data[0][0])
for i in range(len(data)):
output.writeframes(data[i][1])
output.close()

#Deletes all temp files
for i in range(len(textlist)):
os.remove(f"temp audio\temptts{i}.wav")

#Writes the subtitles into an srt file using the durations and textlist
totalduration = 0
srttxt = ''
for counter in range (1, len(durations)):
srttxt = f'{srttxt}{counter}\n'
start = str(datetime.timedelta(seconds=totalduration))
start = start[:start.find('.') + 4].replace('.',',')
totalduration = totalduration + durations[counter]
end = str(datetime.timedelta(seconds = totalduration))
end = end[:end.find('.') + 4].replace('.',',')
srttxt = f'{srttxt}{start} --> {end}\n'
srttxt = f'{srttxt}{textlist[counter - 1]}\n\n'

srttxt = srttxt.replace('0:0 -->', '0:00:00,000 -->')

with open ('srttest.srt','w') as subs:
subs.write(srttxt)
subs.flush()`

I tested it against the whisper method using the sample text in the code above. The whisper model took 1 minute 8 seconds, while the code above took less time at just 49 seconds. I haven't yet tested it with larger text so I don't know if the improvement will still be there.
The one potential drawback of this is that I used nltk to split the text up, which means that the tts may split one of the items in the list again, which may cause an error. I don't think I've seen this happen with my limited testing though, so maybe its a non-issue.

[khaldi-yass]

Hello @arastoof , I'm trying to achieve the same thing, did you succeed into making a good quality audio after splitting the text into chunks? I tried doing that but the AI loses context of the text which results into bad speech tones.

[arastoof]

It's been a while, but I think I remember fixing the issue with the tones by just giving the TTS engine more context between each chunk. I then used https://github.com/linto-ai/whisper-timestamped to get word level timestamps of each chunk, and then used the wave module to cut out any repeated words and stitch together the final product. It isn't really a perfect solution, and is kind of prone to failure, but it worked for my usecase.

[khaldi-yass]

Thank you, based on your answer I got the idea to overlay the joined chunks over a full text audio then based on some tweaking use the chunk timestamps for the subs