[QUESTION] Does exllamav2 support no-dequant inference?

[AaronZLT]

Problem

Given a quant model (for example llama2-7B-nf4), vanilla inference is to dequant the model to fp16 or bf16 to compute, does exllamav2 support no-dequant inference?

[turboderp]

Not sure what you mean? If you use a quantized model, inference will be done using the quantized weights directly. They still have to be converted to FP16 at some point, but this happens in the matmul kernel on individual weights as they're being streamed from VRAM and applied. For long sequences and/or large batch sizes, weights are dequantized one (partial) matrix at a time since that's more efficient.