Android engine/engines for TTS and STT/ASR

[paolo-caroni]

I am writing here because I think the topic is more suitable for discussions than "issues". As requested by several people (see #852 #569 #1088 #569 ), sherpa-onnx would need an engine app for TTS (and a separate or merged one for STT/ASR #580 ). Android offers a standard API for TTS that is currently partially implemented by android .apk packages that support one language at a time and that do not allow the use of multilingual apps (or change language easy).
To save space on devices and implement the choice of language, the engine app should be a single one and allow the language models to be installed separately (without being already present in the package).

An attempt to start the work in this direction was started by @jing332 at the repository but the development stopped at the initial stage.

I would like to understand how I can help in the development of a stable and working app that can reach the level of integration at least equal to eSpeak (I would hope even higher).
Could a fundraiser be useful?

As for the STT/ASR engine the situation is worse. Currently the API is not supported at all. Language detection support DETECTED_LANGUAGE is possible, as it is currently done by a spoken language identification apk that doesn't seem to interface with the API.
Sherpa-onnx seems to have all the capabilities to support language detection, multi-language support, confidence level and more.

Regarding merging or keeping separate TTS and STT, the models are different, so for the same language you would need a model for TTS and a model for STT, so it would probably be easier to have separate engines.

As pointed out here the purpose of the android packages does not seem to be to offer an engine that supports the native android API, but rather to provide examples on how to directly integrate sherpa-onnx in each app with a different API. This would require, if I understood correctly, that each app that listens or spoke to the user should integrate sherpa-onnx, which for obvious reasons makes no sense on Android platform.

Again, I would not be able to write useful code, but I could think about participating in a crowdfunding.

You who have been interested in the topic previously, what do you think?
@mablue @jing332 @csukuangfj @thewh1teagle @domasofan @gyroing

[mablue]

WhT you think about it?
I think the fastest way to reach to the more goals is hide behind tts-server:
jing332/tts-server-android#222

[mablue]

Well, partner, this here Google Accessibility Tools is supposed to be the bee's knees, but it's slower'n a turtle on cold molasses. Especially when we're trying to rope in two language models with that multi sherpa and tts-server rigamarole.
Now, those screen readers like jieshou+ and prodence ain't half bad, but they're a lot of work to set up. It'd be a heap of a lot better if we could just have one app that does it all, like a one-stop shop for accessibility.
And let's face it, a screen reader and a text-to-speech engine are just about the same thing, if ya ask me. It's like a cowboy and his trusty steed. One can't do much without the other. So why not just have one app that can handle both? That way, we could use it with any text-to-speech engine we want, just like that tts-server.
In Android, we got two different ways to do it. One's for the default screen reader, and the other's straight for TalkBack. And guess what? The second one don't care if the screen reader's even turned on or not.

[paolo-caroni]

Your code assume that if there is an alphabetical character the word is english. But I think this is wrong, can't be spanish? or italian? or french? The best way to implement this shouldn't be a language identification? ML Kit is closed source, but fastext-language-identification is open source for example.
This imho should be implemented on the screen reader app, not on the TTS engine.

In any case merge screen reader and TTS engine it's not prohibited, you can do what do you want, but why mantain big code if you can split up? TTS engine is installed and used on all the smartphone, by everyone, but the screen reder is used/installed only in a fraction of them. So as you can see, TTS engine and screen reader are not the same thing. Are like a horse and a cowboy, the cowboy cannot ride without a horse, but a horse can run free or with a jockey instead of a cowboy. Also, I think that the user should decide what engine use, I have tested sherpa onnx on a Huawei Sonic, it's almost unusable, for old/slow smartphone espeak-ng is still the best option.

[mablue]

i said appending language code to end of tts engine files can be usefull to people that want to have two or 3 languages of tts

[powline99]

@paolo-caroni @mablue I am sorry to interrupt your thread but I can't seem to find a solution and perhaps you can help.
I am not advanced enough to get into the coding of this model etc...

I am just trying to get it to work on Android for use in my Accessibility TTS readers. I see there were created apks https://k2-fsa.github.io/sherpa/onnx/tts/apk-engine.html .

When I install on my Google Pixel it only allows one to be used at a time (no input or way to change to a different voice's apk under the app)? It installs the new voice over the same APK from the download.

Is there a easier way to do this with Fdroid app where you can just select the voice you want off the onnx voices?

Thanks,
DD.

[paolo-caroni]

@paolo-caroni @mablue I am sorry to interrupt your thread but I can't seem to find a solution and perhaps you can help. I am not advanced enough to get into the coding of this model etc...

Hi, this is not a issue, but a discudsion, so any request for help is a bit off-topic, also there are issues about that (cited on my first post) have you read them?

I am just trying to get it to work on Android for use in my Accessibility TTS readers. I see there were created apks https://k2-fsa.github.io/sherpa/onnx/tts/apk-engine.html .

When I install on my Google Pixel it only allows one to be used at a time (no input or way to change to a different voice's apk under the app)? It installs the new voice over the same APK from the download.

Is there a easier way to do this with Fdroid app where you can just select the voice you want off the onnx voices?

No, not yet, sadly maybe never, try the jing332 fork https://github.com/jing332/SherpaOnnxTtsEngineAndroid but for me doesn't run as expected.
Obviously it's not yet on fdroid, you have to install it directly from github.