Speed up on CPU with Fast Approximate Matrix Multiplication

[dnhkng]

Hi!

Great work with this project, love it!

As this will probably always be running on a CPU, the project made me recall a paper from a year or so back, on fast approximate matrix multiplication on CPU. It can deliver speed ups 100x the speed of exact matrix multiplication:
https://arxiv.org/abs/2106.10860

There is a C++ implementation here:
https://github.com/dblalock/bolt

Maybe this would get things to GPU speeds on CPU!

[ggerganov]

Interesting stuff! I don't think I can implement this approach - it seems quite complicated. But I got inspired by this idea and started implementing an n-bit quantisation + matrix multiplication in ggml. Still don't know if it will bring any speed-up compared to existing FP16 approaches - will report back when I manage to get it working.

[dnhkng]

Yes, it looks a bit tricky. It seems you need to train a model on each set of layer weights, using the provided python code. The trained model is a like a compressed representation of the matrix.
Then, you use the 'mithral' C++ code with the compressed representation, together with the uncompressed new data from the audio sample, layer by layer. The author also apologies öfter for the code quality, and has said there won't be any updates, which seems a shame.