Support Web Speech API #661
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| import { getTTSEngine } from 'src/apps/chat/store-app-chat'; | ||
| import { CapabilitySpeechSynthesis } from '~/common/components/useCapabilities'; | ||
|
|
||
| import { useCapability as useElevenlabsCapability } from '~/modules/elevenlabs/elevenlabs.client' | ||
| import { speakText as elevenlabsSpeakText } from '~/modules/elevenlabs/elevenlabs.client' | ||
| import { EXPERIMENTAL_speakTextStream as EXPERIMENTAL_elevenlabsSpeakTextStream } from '~/modules/elevenlabs/elevenlabs.client' | ||
|
|
||
| import { useCapability as useBrowserSpeechSynthesisCapability } from '~/modules/browser/speech-synthesis/browser.speechSynthesis.client' | ||
| import { speakText as browserSpeechSynthesisSpeakText } from '~/modules/browser/speech-synthesis/browser.speechSynthesis.client' | ||
| import { EXPERIMENTAL_speakTextStream as EXPERIMENTAL_browserSpeechSynthesisSpeakTextStream } from '~/modules/browser/speech-synthesis/browser.speechSynthesis.client' | ||
|
|
||
| import { useElevenLabsVoices } from '~/modules/elevenlabs/useElevenLabsVoiceDropdown'; | ||
| import { useBrowserSpeechVoices } from '~/modules/browser/speech-synthesis/useBrowserSpeechVoiceDropdown'; | ||
|
|
||
| export const TTSEngineList: string[] = [ | ||
|
There was a problem hiding this comment. See other note on this. I wonder if this would be better served as a map: export type TTSEngineKey = 'elevenlabs' | 'webspeech';
const TTSEngineList: { [key in TTSEngineKey]: { label: string, priority: number } } = {
'elevenlabs': {
label: 'ElevenLabs',
priority: 2,
},
'webspeech': {
label: 'Web Speech API',
priority: 1,
},
};I've added an attribute called 'priority' to show how we can extend it in the future, for instance with default configurations. There was a problem hiding this comment. After a try. I think use map for extending is a good idea, but i suggest use a list contained map: export type TTSEngineKey = 'Elevenlabs' | 'Web Speech API';
export type ASREngineKey = 'Web Speech API';
export const TTSEngineList: { key: TTSEngineKey; label: string }[] = [
{
key: 'Elevenlabs',
label: 'ElevenLabs',
},
{
key: 'Web Speech API',
label: 'Web Speech API',
},
];because there are only 2 case using the variable now, one is |
||
| 'Elevenlabs', | ||
| 'Web Speech API' | ||
| ] | ||
|
|
||
| export const ASREngineList: string[] = [ | ||
| 'Web Speech API' | ||
| ] | ||
|
|
||
| export function getConditionalVoices(){ | ||
| const TTSEngine = getTTSEngine(); | ||
| if (TTSEngine === 'Elevenlabs') { | ||
| return useElevenLabsVoices | ||
| }else if (TTSEngine === 'Web Speech API') { | ||
| return useBrowserSpeechVoices | ||
| } | ||
| throw new Error('TTSEngine is not found'); | ||
| } | ||
|
|
||
| export function hasVoices(): boolean { | ||
| console.log('getConditionalVoices', getConditionalVoices()().hasVoices) | ||
| return getConditionalVoices()().hasVoices; | ||
| } | ||
|
|
||
| export function getConditionalCapability(): () => CapabilitySpeechSynthesis { | ||
| const TTSEngine = getTTSEngine(); | ||
| if (TTSEngine === 'Elevenlabs') { | ||
| return useElevenlabsCapability | ||
| }else if (TTSEngine === 'Web Speech API') { | ||
| return useBrowserSpeechSynthesisCapability | ||
| } | ||
| throw new Error('TTSEngine is not found'); | ||
| } | ||
|
|
||
| export function useCapability(): CapabilitySpeechSynthesis { | ||
|
There was a problem hiding this comment. Crash issue identified to this hook (the one that gave the black screen in the screenshot). Seems that when switching provider, there's a react out-of-order issue. Only when switching TTS providers I believe. There was a problem hiding this comment. It's possible that to fix this properly, we may have to overhaul the ttsengine reactivity (hooks) |
||
| return getConditionalCapability()(); | ||
| } | ||
|
|
||
|
|
||
| export async function speakText(text: string, voiceId?: string) { | ||
| const TTSEngine = getTTSEngine(); | ||
| if (TTSEngine === 'Elevenlabs') { | ||
| return await elevenlabsSpeakText(text, voiceId); | ||
|
There was a problem hiding this comment. This is a way of doing it. I wonder if we should have a common Interface, and have all the TTS providers (e.g. OpenAI has 2 as well, one via their TTS and one in the new [audio] models), and there's Play.ht and more. A way to reduce the switch-cases (or ifs) will be to have a common interface, such as ISpeechSynthesis, that will be returned by getTTSEngine (basically instead of the string it would return an object that implements the interface). Just a thought. |
||
| }else if (TTSEngine === 'Web Speech API') { | ||
| return await browserSpeechSynthesisSpeakText(text, voiceId); | ||
| } | ||
| throw new Error('TTSEngine is not found'); | ||
| } | ||
|
|
||
| // let liveAudioPlayer: LiveAudioPlayer | undefined = undefined; | ||
|
|
||
| export async function EXPERIMENTAL_speakTextStream(text: string, voiceId?: string) { | ||
| const TTSEngine = getTTSEngine(); | ||
| if (TTSEngine === 'Elevenlabs') { | ||
| return await EXPERIMENTAL_elevenlabsSpeakTextStream(text, voiceId); | ||
| }else if (TTSEngine === 'Web Speech API') { | ||
| return await EXPERIMENTAL_browserSpeechSynthesisSpeakTextStream(text, voiceId); | ||
| } | ||
| throw new Error('TTSEngine is not found'); | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I love this.