CI: test-backend-ops -j8

[jeffbolznv]

Overview

Try enabling test-backend-ops -j 8 in CI. Let's see what happens.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, written by codex.

[ggerganov]

I think the proper way would be to read the number of threads from an env variable and set it correctly in the workflows via $(nproc) for example. Or alternatively, treat -j -1 as std::thread::hardware_concurrency().

[jeffbolznv]

I mostly wanted to see if any CI would fail with multiple threads (but some other flakiness happened instead). I agree this is not the final shape this should be in.

[jeffbolznv]

Linux-intel-vulkan crashes pretty quickly with heap corruption, possible driver bug?

mac-vulkan crashes with a devicelost. Likely a moltenvk issue...

vulkan-llvmpipe times out, I reenabled the full test-backend-ops but only gave it a 30m timeout. I'll be curious to see if this does better after #23641.

[ggerganov]

Btw, the self-hosted coompat2 job seems to be consistently about 2x slower than the respective coopmat job:

https://github.com/ggml-org/llama.cpp/actions/runs/28128061386
https://github.com/ggml-org/llama.cpp/actions/runs/28118398923
https://github.com/ggml-org/llama.cpp/actions/runs/28113462160

Is this expected, or do you see some misconfiguration for these vulkan jobs? cc @ggml-org/ggml-vulkan

[jeffbolznv]

The runtime difference is bigger than I would have expected:

KHR_coopmat: 2026-06-24T20:52:49.8165727Z 38/45 Test #38: test-backend-ops .................. Passed 213.86 sec
NV_coopmat2: 2026-06-24T21:43:10.1477278Z 38/45 Test #38: test-backend-ops .................. Passed 905.31 sec

When I run these locally I get:

NV_coopmat2 with cold shader cache: 570s
KHR_coopmat1 with cold shader cache: 376s
KHR_coopmat1 with warm shader cache: 168s

I suspect the cm2 test system it not getting shader caching, but the cm1 test system is.

The shader cache should be in (roughly) $HOME/.cache/nvidia/GLCache/.... If the system didn't have permission to write this location, it could explain it. Or it's possible the cache has become full and isn't able to make space for some reason, in which case I'd suggest to start by deleting the cache files.