Performance degradation on Vulkan when increasing -ngl value?

[stduhpf]

I don't know if it's a bug or just a quirk, but I get some surprising results when benchmarking the performance of the model when offloading a different number of layers to my AMD GPU with Vulkan. As expected, performance scales nicely with the number of layers offloaded until the 31st one. Then the prompt processing performance falls off abruptly. The same thing happens with token generation performance at 37 layers offloaded.

.\build\bin\Release\llama-bench.exe -m .\models\Mistral\nemo\ggml-model-Q4_K_M.gguf -ngl 20,25,30,31,32,34,36,37,38,40,41 -t 12

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 5700 XT (AMD proprietary driver) | uma: 0 | fp16: 1 | warp size: 64

model	size	params	backend	ngl	test	t/s
ggml_vulkan: Compiling shaders..............................Done!
llama 13B Q4_K - Medium	6.96 GiB	12.25 B	Vulkan,RPC	20	pp512	191.57 ± 0.22
llama 13B Q4_K - Medium	6.96 GiB	12.25 B	Vulkan,RPC	20	tg128	9.72 ± 0.06
llama 13B Q4_K - Medium	6.96 GiB	12.25 B	Vulkan,RPC	25	pp512	196.20 ± 1.12
llama 13B Q4_K - Medium	6.96 GiB	12.25 B	Vulkan,RPC	25	tg128	11.41 ± 0.06
llama 13B Q4_K - Medium	6.96 GiB	12.25 B	Vulkan,RPC	30	pp512	202.64 ± 0.45
llama 13B Q4_K - Medium	6.96 GiB	12.25 B	Vulkan,RPC	30	tg128	13.93 ± 0.04
llama 13B Q4_K - Medium	6.96 GiB	12.25 B	Vulkan,RPC	31	pp512	203.80 ± 0.19
llama 13B Q4_K - Medium	6.96 GiB	12.25 B	Vulkan,RPC	31	tg128	14.22 ± 0.18
llama 13B Q4_K - Medium	6.96 GiB	12.25 B	Vulkan,RPC	32	pp512	44.00 ± 2.07
llama 13B Q4_K - Medium	6.96 GiB	12.25 B	Vulkan,RPC	32	tg128	15.23 ± 0.04
llama 13B Q4_K - Medium	6.96 GiB	12.25 B	Vulkan,RPC	34	pp512	48.40 ± 0.15
llama 13B Q4_K - Medium	6.96 GiB	12.25 B	Vulkan,RPC	34	tg128	15.84 ± 0.05
llama 13B Q4_K - Medium	6.96 GiB	12.25 B	Vulkan,RPC	36	pp512	50.84 ± 1.58
llama 13B Q4_K - Medium	6.96 GiB	12.25 B	Vulkan,RPC	36	tg128	17.53 ± 0.03
llama 13B Q4_K - Medium	6.96 GiB	12.25 B	Vulkan,RPC	37	pp512	72.70 ± 0.36
llama 13B Q4_K - Medium	6.96 GiB	12.25 B	Vulkan,RPC	37	tg128	5.59 ± 0.01
llama 13B Q4_K - Medium	6.96 GiB	12.25 B	Vulkan,RPC	38	pp512	67.28 ± 0.40
llama 13B Q4_K - Medium	6.96 GiB	12.25 B	Vulkan,RPC	38	tg128	5.28 ± 0.04
llama 13B Q4_K - Medium	6.96 GiB	12.25 B	Vulkan,RPC	40	pp512	59.34 ± 1.58
llama 13B Q4_K - Medium	6.96 GiB	12.25 B	Vulkan,RPC	40	tg128	5.06 ± 0.01
llama 13B Q4_K - Medium	6.96 GiB	12.25 B	Vulkan,RPC	41	pp512	62.03 ± 0.11
llama 13B Q4_K - Medium	6.96 GiB	12.25 B	Vulkan,RPC	41	tg128	5.37 ± 0.02

Anyone knows what could be causing such a significant jump in performance?

[ExtReMLapin]

Could it be OS related, like no more vram and it switches to ram ?

[stduhpf]

Could be, but windows task manager still reports the VRAM as not completely full, which is why I'm confused. Also, when I get out of VRAM with the Vulkan backend, it usually just crashes the app.