MoE model decode hangs on Jetson Orin AGX (SM87) since b7309

[BusinessBuilders]

Bug Description

MoE model inference (decode/warmup) hangs indefinitely on NVIDIA Jetson Orin AGX (SM87, compute capability 8.7) on all commits after b7309 (017761daf, Dec 6 2025). The model loads successfully but decode never produces tokens — the CUDA kernel appears to hang.

This affects all MoE models tested, not just one specific model or quantization:

• GLM-4.7-Flash Q8_0 (Unsloth, DeepSeek2 architecture, 64x2.6B MoE, 3B active)
• Qwen3-Coder-30B-A3B Q4_K_M (Qwen3 MoE architecture, 30B params, 3B active)

Both models work perfectly on b7309 (017761daf) but hang on latest master (bd90fc74c).

Environment

• Device: NVIDIA Jetson Orin AGX 64GB (unified memory)
• GPU: Orin (SM87, compute capability 8.7)
• JetPack: R36.4.7
• CUDA: 12.6
• Build flags: -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=87 -DGGML_CUDA_FA_ALL_QUANTS=ON -DGGML_CUDA_F16=ON -DCMAKE_BUILD_TYPE=Release

How to Reproduce

# Build latest master
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=87 -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)

# This hangs indefinitely (never produces tokens):
build/bin/llama-server \
  --model GLM-4.7-Flash-Q8_0.gguf \
  -ngl 99 --ctx-size 8192 --port 11434 --host 0.0.0.0 --jinja -np 1 \
  --override-kv deepseek2.expert_gating_func=int:2

# Server starts, model loads, but hangs at:
# "warming up the model with an empty run"
# Using --no-warmup, the server starts but decode requests hang forever.

# Same behavior with Qwen3-Coder-30B-A3B Q4_K_M

Bisect Results

We performed a git bisect between 017761daf (good, b7309) and bd90fc74c (bad, master HEAD).

Finding 1: llama-cli decode hang

First bad commit: c8554b66e — "graph: use fill instead of scale_bias in grouped expert selection (#17867)"

- selection_probs = ggml_set_rows(ctx0, ggml_scale_bias(ctx0, selection_groups, 0.0f, -INFINITY), ...);
+ selection_probs = ggml_set_rows(ctx0, ggml_fill(ctx0, selection_groups, -INFINITY), ...);

This 1-line change replaces ggml_scale_bias(tensor, 0.0, -INFINITY) with ggml_fill(tensor, -INFINITY) for masking non-selected expert groups. Both are mathematically equivalent, but the ggml_fill path hangs during CUDA execution on SM87.

Finding 2: llama-server warmup still hangs even with revert

Reverting c8554b66e and rebuilding llama-server from master does not fix the hang. The server still hangs at warmup (which performs a llama_decode). This indicates there are additional breaking changes in master's decode/graph execution path for MoE models on SM87.

Finding 3: llama-cli loading hang

First bad commit: 6c2131773 — "cli: new CLI experience (#17824)"

The new llama-cli binary hangs during model loading on SM87. This only affects the new CLI, not llama-server.

Working Configuration

On b7309 (017761daf), both models run perfectly:

• GLM-4.7-Flash Q8_0: 21.8 tok/s generation, 37.9 tok/s prompt processing
• Qwen3-Coder-30B-A3B Q4_K_M: 37.1 tok/s generation, 57.2 tok/s prompt processing

Both with 100k context, all layers on GPU (-ngl 99), Flash Attention enabled.

Summary

Multiple commits between b7309 and master break MoE model decode on Jetson Orin (SM87). The most identifiable is c8554b66e which changes the expert selection masking from ggml_scale_bias to ggml_fill, but this is not the only breaking change — additional changes in the graph/backend/CUDA execution paths also contribute to the hang.

This may affect other SM87 devices and potentially other MoE models. Dense models were not tested.

[BusinessBuilders]

Update: Server-specific bisect completed

The original bisect used llama-cli which has different code paths. A second bisect using llama-server found the actual breaking commit for the server hang:

First bad commit: a83c73a — "[CUDA] Reduce CPU-side stalls due to the CUDA command buffer being full (#19042)"

This commit adds setenv("CUDA_SCALE_LAUNCH_QUEUES", "4x", 0) in ggml_backend_cuda_reg() to scale up the CUDA command buffer for multi-GPU pipeline parallelism.

On Jetson Orin AGX (SM87, unified memory), this causes llama-server to hang indefinitely during warmup decode. The model loads fine, the server starts listening, but the first llama_decode() call never returns.

Workaround

Setting CUDA_SCALE_LAUNCH_QUEUES=1x before launching the server completely fixes the issue:

export CUDA_SCALE_LAUNCH_QUEUES=1x
llama-server --model model.gguf -ngl 99 ...

Since the code uses setenv(..., 0) (no overwrite), pre-setting the env var neutralizes it.

Verified

• Latest master (bd90fc74c) with CUDA_SCALE_LAUNCH_QUEUES=1x: 20.7 tok/s generation, works perfectly
• Latest master without workaround: hangs at warmup decode indefinitely
• Affects both GLM-4.7-Flash (DeepSeek2 MoE) and Qwen3-Coder-30B-A3B (Qwen3Moe)
• Jetson Orin AGX 64GB, CUDA 12.6, SM87

Suggestion

The CUDA_SCALE_LAUNCH_QUEUES scaling should probably be gated behind a multi-GPU check or made opt-in, since it breaks single-GPU unified memory systems (Jetson).

CC @gaugarg-nvidia

[ggerganov]

cc @gaugarg-nv - Not sure how reliable this report is, but might be something worth checking.

[gaugarg-nv]

Thanks for the report. I'm trying to get access to a system where I can repro this.

The CUDA_SCALE_LAUNCH_QUEUES scaling should probably be gated behind a multi-GPU check or made opt-in, since it breaks single-GPU unified memory systems (Jetson).

The environment variable needs to be set before we can make any CUDA API call. So, setting the env variable after using the CUDA API to detect the number of CUDA devices won't work.
In any case, if this is causing regression, I agree that this should be made opt-in.

@ggerganov I have reverted the change with this PR for now: #19227

I will try to reproduce this once I get access to the Jetson system and try to come up with a better solution if this reproduces.

[ggerganov]

Ok, I think there is no rush to merge #19227 - let's take some time to repro and confirm the issue. Thanks.

[elfarolab]

Hello,

I am running llama.cpp (88d23ad) b7860 on Jetson AGX Orin.
Model: Qwen3-30B-A3B-Instruct, Q4_K_M. MoE
Fully loaded into RAM. Not running any other big processes.

I am not having this issue at my side.

Thank you.

[BusinessBuilders]

@gaugarg-nv @ggerganov Full reproduction details and a fix PR:

Environment

Component	Version
Device	Jetson AGX Orin 64GB
JetPack / L4T	R36.4.7 (GCID 42132812, Sep 18 2025)
Kernel	5.15.148-tegra
NVIDIA Driver	540.4.0
CUDA Toolkit	12.6.2 (V12.6.77)
CUDA API	12060
cuDNN	9.4.0.58
GCC/G++	11.4.0
CMake	4.1.0

Build flags

cmake -B build -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=87 \
  -DGGML_CUDA_FA_ALL_QUANTS=ON \
  -DGGML_CUDA_F16=ON \
  -DCMAKE_BUILD_TYPE=Release

Bisect

9-step bisect using llama-server (not llama-cli):

• Last good: fc3cdf32c
• First bad: a83c73a18

Fix PR

#19230 — Detects Tegra via /sys/devices/soc0/family and skips the launch queue scaling. Tested on Jetson: 20.4 tok/s, no deadlock. Alternative to full revert #19227.

@elfarolab — Could you share your build flags (specifically GGML_CUDA_FA_ALL_QUANTS and GGML_CUDA_F16), JetPack version, and nvidia-smi output? May help narrow down why you're not hitting this.

[elfarolab]

@BusinessBuilders

sure, I am using a parametric automatic script to build llama.cpp, because it builds for different architectures, some options are obvious/redundant and some others are not important, also about nvidia-smi, I deeply modified the Jetson AGX Orin to clean as much as possible RAM, squeezing all possible resources to make room to llama.cpp so nvidia-smi is not working anymore, it just print empty values..

CMAKE_BUILD_TYPE=Release
CMAKE_INSTALL_PREFIX=$LLAMACPP_PREFIX_DIR
CMAKE_CUDA_COMPILER=/usr/local/cuda-12.6/bin/nvcc
CMAKE_CUDA_ARCHITECTURES=87
GGML_CUDA=ON
GGML_CUDA_FA=ON
GGML_CUDA_GRAPHS=ON
GGML_CUDA_FORCE_CUBLAS=ON
GGML_BLAS=ON
GGML_BLAS_VENDOR=OpenBLAS
#GGML_BLAS_VENDOR=FLAME
#BLAS_INCLUDE_DIRS=/opt/usbhd/SRC/blis_install/include
#BLAS_ROOT=/opt/usbhd/SRC/blis_install
BLAS_LIBRARIES="$OPENBLAS_LIB"
#BLAS_LIBRARIES=/opt/usbhd/SRC/blis_install/lib
#GGML_BLAS_USE_BLIS=ON
GGML_BLAS_USE_OPENBLAS=ON
GGML_CUDA_USE_MMQ=OFF
##GGML_CUDA_USE_MMQ=ON
GGML_CUDA_FA_ALL_QUANTS=ON
#GGML_AVX=OFF
#GGML_AVX2=OFF
#GGML_AVX512=OFF
#GGML_SSE42=OFF
#GGML_F16C=OFF
#GGML_FMA=OFF
GGML_ACCELERATE=OFF
GGML_METAL=OFF
GGML_OPENCL=OFF
GGML_SYCL=OFF
GGML_HEXAGON=OFF
GGML_HIP=OFF
GGML_WEBGPU=OFF
GGML_VULKAN=OFF
GGML_LTO=ON
BUILD_SHARED_LIBS=OFF
GGML_STATIC=ON
GGML_CUDA_F16=ON
#GGML_CUDA_BF16=ON
BLA_STATIC=ON
LLAMA_BUILD_COMMON=ON
LLAMA_BUILD_EXAMPLES=ON
LLAMA_BUILD_TESTS=OFF
LLAMA_BUILD_TOOLS=ON
LLAMA_BUILD_SERVER=ON
LLAMA_OPENSSL=OFF
LLAMA_CURL=OFF
GGML_CUDA_JETSON_DEVICE=ON
GGML_CUDA_ENABLE_UNIFIED_MEMORY=ON
LLAMA_TOOLS_INSTALL=ON
GGML_BACKEND_DL=OFF
GGML_CPU_ALL_VARIANTS=OFF
#DGGML_SCHED_NO_REALLOC=ON
GGML_CUDA_NO_VMM=ON

[BusinessBuilders]

strace of the deadlock

Ran build_master (unpatched, CUDA_SCALE_LAUNCH_QUEUES=4x) under strace -e trace=ioctl,futex -f -t -T on Jetson AGX Orin 64GB.

What happens

Thread 182733 (CUDA kernel submission) calls ioctl(5, DRM_IOCTL_RADEON_GEM_PWRITE) to push commands to the GPU command buffer:

1. 7487 successful submissions — returns = 0, each taking ~1-4ms
2. Then EAGAIN loop — at 13:11:26, the same ioctl starts returning EAGAIN (resource temporarily unavailable) every ~104ms, indefinitely
3. No other threads make progress — server health thread just does futex waits with 500ms timeouts

Transition point (lines 9970-9982)

182733 13:11:26 ioctl(5, DRM_IOCTL_RADEON_GEM_PWRITE, 0xffffeec8f548) = 0 <0.000568>    ← last success
182733 13:11:26 ioctl(3, _IOC(..., 0x4e, ...)) = 0                                       ← NVIDIA RM ioctl
182733 13:11:26 ioctl(4, _IOC(..., 0x47, ...)) = 0                                       ← NVIDIA UVM ioctl
182733 13:11:26 ioctl(7, _IOC(..., 0x41, ...)) = 0                                       ← NVIDIA ctrl ioctl
182733 13:11:26 ioctl(5, DRM_IOCTL_RADEON_GEM_PWRITE, 0xffffeec8f628) = -1 EAGAIN        ← deadlock starts
182733 13:11:26 ioctl(5, DRM_IOCTL_RADEON_GEM_PWRITE, 0xffffeec8f628) = -1 EAGAIN        ← loops forever

Interpretation

The GPU command buffer (DRM fd 5) fills up and never drains. With 4x scaling the buffer is larger, so more MMQ kernel launches accumulate. The MMQ path dispatches many small kernels per MoE expert (the expert loop in ggml_cuda_mul_mat_id interleaves kernel launches with cudaStreamSynchronize). Once the buffer is full, the CPU can't submit the sync command, and the GPU can't drain the buffer because it needs that sync to make progress.

FORCE_CUBLAS=ON avoids this because cuBLAS batches its kernel submissions differently and doesn't hit the buffer-full condition.

Why Jetson but not desktop single-GPU

On desktop GPUs the command buffer lives in dedicated VRAM with its own memory controller. On Jetson's unified memory (CPU and GPU share 64GB, 204 GB/s), the DRM_IOCTL_RADEON_GEM_PWRITE path for writing to the command buffer shares the same memory bus as CPU operations. The 4x buffer scaling creates a condition where the buffer fills before the GPU can start draining it.

@DocShotgun confirmed no issues on desktop RTX Pro 6000 single-GPU — this is Jetson/unified-memory specific, not a general single-GPU issue. Apologies for the incorrect framing in the earlier PR.

Numbers

• Successful DRM PWRITEs before deadlock: 7487
• EAGAIN retries until timeout: 238+ (~104ms each)
• Only thread 182733 was blocked on GPU ops

---

Co-authored-by: Claude (Anthropic) — strace capture and analysis

[gaugarg-nv]

Once the buffer is full, the CPU can't submit the sync command, and the GPU can't drain the buffer because it needs that sync to make progress.

I don't think this is correct. GPUs don't need a sync to start draining from the command buffer.
@elfarolab would it be possible for you to reproduce this with the options provided by @BusinessBuilders ?

@BusinessBuilders It seems you might be hitting an inherent issue that is just getting repro'd with CUDA_SCALE_LAUNCH_QUEUES=4x. Can you file a bug on Nvidia forum?

[elfarolab]

@gaugarg-nv
I will try right now

[BusinessBuilders]

@gaugarg-nv I will try right now

-DGGML_CUDA_FORCE_CUBLAS=ON

If you compile WITHOUT that flag it should reproduce @elfarolab

TY