Using LQER to improve low-bit quants

[compilade]

Since the recent LoRA refactor by @ngxson in #8332, I think it should be possible to improve existing quantization schemes with Low-Rank Quantization Error Reconstruction (see https://arxiv.org/abs/2402.02446)

It would only need two things:

1. Some script to make a LoRA out of the differences of two GGUF files of the same model for tensors which can use LoRA
   • If in Python, dequantization functions would need to be written
     • I will refactor gguf-py/gguf/quants.py to make this easier.
   • If in C++, we'd need to find a way to do SVD (Singular Value Decomposition)
2. Allowing to store a LoRA adapter along with the model weights in a single model file
   • Do we also need some extra metadata to signal that a model file (with dense weights) has a LoRA adapter inside?

And also I think L²QER could be implemented with the existing imatrix files.

[ngxson]

I read the paper but it's still not very clear for me if L²QER change something at inference time or not.

Please correct if I'm wrong, but what I understand is that we firstly do LQER by running SVD to get LoRA, then further optimizing it via L²QER (around page 6 of the paper).

Anyway,

If in Python, dequantization functions would need to be written

FYI, someone already done the python implementation of GGML quants in HF transformers library. Hopefully this can help your implementation.

If in C++, we'd need to find a way to do SVD (Singular Value Decomposition)

I'm really interested in this, because control vector can be benefit from SVD (we're currently using PCA)

Do we also need some extra metadata to signal that a model file (with dense weights) has a LoRA adapter inside?

Yes, I think it would be useful to know if we should call llama_lora_adapter_init (or maybe a refactored version of it)

The current llama_lora_adapter_init function does not use mmap at all, that need to be fixed before we can proceed. Also, lora with split (shard) model is not supported atm. Hopefully we can bring more gguf-reader logic (i.e. with mmap & shard support) into ggml library, so it can be reused anywhere.

[compilade]

I read the paper but it's still not very clear for me if L²QER change something at inference time or not.

Please correct if I'm wrong, but what I understand is that we firstly do LQER by running SVD to get LoRA, then further optimizing it via L²QER (around page 6 of the paper).

From my understanding of Section 3.2, L²QER requires to scale the error by $S$ before SVD, and then $S$ is cancelled out when making A and B. $S$ comes from the activations (described in Appendix A), and this looks similar to what imatrix does (except that imatrix seems to do a mean of sums of squares of the activations (I think?) instead of a max of the absolute means), but I think imatrix data could maybe still be used for $S$.

LQER seems to be simply doing SVD of the quantization error, so this will be easier to implement first.

Apart from the LoRA tensors (which are already supported because of #8332), I don't think this changes anything at inference time.

If in C++, we'd need to find a way to do SVD (Singular Value Decomposition)

I'm really interested in this, because control vector can be benefit from SVD (we're currently using PCA)

There's a test in ggml with SVD, but I'm not sure if it works, because it seems to use the sgesvd_ function out of nowhere.

FYI, someone already done the python implementation of GGML quants in HF transformers library. Hopefully this can help your implementation.

Nice! This seems to come from https://github.com/99991/pygguf. But I'll likely re-implement the dequantization anyway to better fit how gguf-py/gguf/quants.py will be structured as in #8838, and also to better understand their structure to be able to possibly eventually write documentation about that (since @mofosyne seems to be interested in this (ref: #8151 (comment))).

[ngxson]

There's a test in ggml with SVD, but I'm not sure if it works, because it seems to use the sgesvd_ function out of nowhere

Actually, the test file you mentioned only runs on ARM: ggerganov/ggml@3b3ad42

Indeed, a quick google search shows that sgesvd_ is a function provided by ARM: https://developer.arm.com/documentation/101004/2404/LAPACK-Linear-Algebra-Package/LAPACK-singular-value-decompositions-routines/sgesvd

In anyway, I think we can re-write our own implementation in cpp using ggml. I will try on my side by following this python code, but feel free to tell if you have other suggestions.

[jukofyork]

In anyway, I think we can re-write our own implementation in cpp using ggml. I will try on my side by following this python code, but feel free to tell if you have other suggestions.

That method is only useful if you want the first couple of singular vectors - the numerical errors will compound massively after.

https://people.duke.edu/~hpgavin/SystemID/References/Golub+Reinsch-NM-1970.pdf

is the standard algorithm used in most software. The ALGOL code at the end of the paper has been (auto) converted numerous times to FORTRAN and C, but the most understandable is probably from the GSL:

https://github.com/ampl/gsl/blob/master/linalg/svd.c

I'm not sure how the GPL-3 and MIT licenses work together, but it's often fairly easy to just lift a function from the GSL source and get it working in isolation.

[jukofyork]

Here's a better explanation:

https://people.inf.ethz.ch/gander/talks/Vortrag2022.pdf

[jukofyork]

If it's any use then I tidied up the Mergekit code to extract LoRAs from fine turned models:

arcee-ai/mergekit#333

It works on all but the weirdest models now (eg: mistral-v0.3 prepended new tokens!?). I successfully managed to take full rank LoRAs and reconstructs again for a few test cases, so it definitely is working and seems quite numerically stable now the sqrt(S) is distributed.

Might be useful to make a reference version using pytorch and then compare with a native C/C++ reimplemention.

---

EDIT: The only part of Mergekit it's using is the "lazy tensor loading" and other than that it's mostly based on:

https://github.com/thomasgauthier/LoRD

(I'm not sure how it ended up part of Mergekit)

[compilade]

Nice! This is useful.

And the best part is that I think most of the problems you mentioned will not exist when doing LQER on GGUF models, because between a full-precision model and its quantizations,

• The tensor names are always the same
• The tensor dimensions are always the same

There are basically (almost) no edge cases.

Distributing sqrt(S) seems like a good idea, thanks for mentioning it.

[ggerganov]

Interesting work, will be cool to try to implement this approach and see how the perplexity improves for different ranks

I'm looking at some of the results in the paper and not sure how to interpret Appendix B:

Based on the graph, it seems L2QER performs worse (i.e. higher error) compared to LQER, while the text states the opposite. Am I reading it wrong?