Google Cloud Text-to-speech word timestamps

18
votes

I'm generating speech through Google Cloud's text-to-speech API and I'd like to highlight words as they are spoken.

Is there a way of getting timestamps for spoken words or sentences?

2
text-to-speechspeech-synthesisgoogle-text-to-speech
I don't think you can do this with google cloud yet but if you using an android device and the google TextToSpeech engine, you can do this: stackoverflow.com/questions/59488998/… - Nerdy Bunz
You can break the sentences into words as tokens and create or hightlight the words by your own code. You also have to config the settings properly and maybe have to use thread for for sending multiple words at a same time. Can you please share the code. - Akash Badam

2 Answers

2
votes

This question seems to have gotten quite popular so I thought I'd share what I ended up doing. This method will probably only work with English or similar languages.

I first split text on any punctuation that causes a break in speaking. Each "sentence" is converted to speech separately. The resulting audio files have a seemingly random amount of silence at the end which needs to be removed before joining them, this can be done with the FFmpeg silencedetect filter. You can then join the audio files with an appropriate gap. Approximate word timestamps can be linearly interpolated within the sentences.

2
votes

You can do this using SSML and v1beta1 version of Google Cloud's text-to-speech API: https://cloud.google.com/text-to-speech/docs/reference/rest/v1beta1/text/synthesize#TimepointType

  1. Add <mark> SSML tags to the point in the text that you want a timestamp for (maybe at the end of each sentence).
  2. Set TimepointType to SSML_MARK. If this field is not set, timepoints are not returned by default.