Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Really slow transcription on Android platform. #47

Open
codyrowland opened this issue Aug 4, 2023 · 5 comments
Open

Really slow transcription on Android platform. #47

codyrowland opened this issue Aug 4, 2023 · 5 comments
Labels
enhancement New feature or request

Comments

@codyrowland
Copy link

Hello,

I was recently testing this on Android and was noticing a large delay in transcription. I already tried creating a development build and adjusting the settings but got the same results. The best I could get was about 10000ms for a really short transcription. I ran a deep profile on it and noticed there was a huge delay coming from one of the 4 threads. I'm testing this on a Quest 2. It's coming from the WhisperNative.whisper_full function. Any ideas on what this could be from? There seems to be a lot of GC as well. Appreciate any pointers!

Unity_FS4dnfVzDj

@codyrowland codyrowland changed the title Really slow Inference on Android platform. Really slow transcription on Android platform. Aug 4, 2023
@Macoron
Copy link
Owner

Macoron commented Aug 4, 2023

Sadly, Android was always black sheep of whisper.cpp. Check this issues:
ggerganov/whisper.cpp#1070
ggerganov/whisper.cpp#1022
ggerganov/whisper.cpp#959

I read that some folks tried to use it on Quest 2 and also have mixed result.

The only thing that I recommend is to try play with settings. The most obvious one is enable SpeedUp that you can find in WhisperManager.

Also you can try to play around with a number of threads that you allocate for whisper. Right now it forced to use single thread, but maybe you can find some other which will work better on a quest.

// for some reason on android one thread works
// 10x faster than multithreading
#if UNITY_ANDROID && !UNITY_EDITOR
param.ThreadsCount = 1;
#endif
return param;

Please write back if you will have any results, I'm really interest to hear if it can run faster on quests.

@Macoron Macoron added the bug Something isn't working label Aug 4, 2023
@codyrowland
Copy link
Author

We ended up setting the threads to 3 and enabling SpeedUp and the time went down to about 7000ms. This was a pretty large improvement for us and will most likely work for our use case. I'll try to improve that and report back with what I find.

@Macoron
Copy link
Owner

Macoron commented Aug 14, 2023

We ended up setting the threads to 3 and enabling SpeedUp and the time went down to about 7000ms. This was a pretty large improvement for us and will most likely work for our use case. I'll try to improve that and report back with what I find.

Glad to hear that. Hope guys from whisper.cpp will improve Android performance in future.

I'll keep this issue open for better discoverability.

@Macoron Macoron added enhancement New feature or request and removed bug Something isn't working labels Aug 14, 2023
@Theadmaster
Copy link

did you get an improvement on Android?

@codyrowland
Copy link
Author

codyrowland commented Oct 27, 2023

I haven't had the time to explore potential solutions to this. However, I think the solution would have to be handled with Whisper.cpp. Looks like there have been mentions of GPU accelerated solutions. (ggerganov/whisper.cpp#1070)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants