Welcome to WhisperFusion. WhisperFusion builds upon the capabilities of the WhisperLive and WhisperSpeech by integrating Mistral, a Large Language Model (LLM), on top of the real-time speech-to-text pipeline. Both LLM and Whisper are optimized to run efficiently as TensorRT engines, maximizing performance and real-time processing capabilities. While WhiperSpeech is optimized with torch.compile.
-
Real-Time Speech-to-Text: Utilizes OpenAI WhisperLive to convert spoken language into text in real-time.
-
Large Language Model Integration: Adds Mistral, a Large Language Model, to enhance the understanding and context of the transcribed text.
-
TensorRT Optimization: Both LLM and Whisper are optimized to run as TensorRT engines, ensuring high-performance and low-latency processing.
-
torch.compile: WhisperSpeech uses torch.compile to speed up inference which makes PyTorch code run faster by JIT-compiling PyTorch code into optimized kernels.
- A GPU with at least 24GB of RAM
- For optimal latency, the GPU should have a similar FP16 (half) TFLOPS as the RTX 4090. Here are the hardware specifications for the RTX 4090.
The demo was run on a single RTX 4090 GPU. WhisperFusion uses the Nvidia TensorRT-LLM library for CUDA optimized versions of popular LLM models. TensorRT-LLM supports multiple GPUs, so it should be possible to run WhisperFusion for even better performance on multiple GPUs.
- We provide a pre-built TensorRT-LLM docker container that has both whisper and phi converted to TensorRT engines and WhisperSpeech model is pre-downloaded to quickly start interacting with WhisperFusion.
docker run --gpus all --shm-size 64G -p 6006:6006 -p 8888:8888 -it ghcr.io/collabora/whisperfusion:latest- Start Web GUI
cd examples/chatbot/html
python -m http.server- We provide the docker image for cuda-architecures 89 and 90. If you have a GPU with a different cuda architecture. For e.g. to build for RTX 3090 with cuda- architecture 86
bash build.sh 86-realThis should build the ghcr.io/collabora/whisperfusion:latest for RTX 3090.
For questions or issues, please open an issue. Contact us at: [email protected], [email protected], [email protected]