Eval bug: Jinja fails on `gpt-oss-120b` when using Vulkan

[danielholanda]

Description

Jinja fails on gpt-oss-120b when using Vulkan.

Name and Version

version: 6097 (9515c61)
built with clang version 19.1.5 for x86_64-pc-windows-msvc

Operating systems

Windows 11

GGML backends

Vulkan

Hardware

AMD RyzenAI MAX+ 395 w/ Radeon 8060S (Strix Halo)

Models

gpt-oss-120b-Q4_K_M
https://huggingface.co/unsloth/gpt-oss-120b-GGUF

Problem description & steps to reproduce

To reproduce the issue, simply download unsloth/gpt-oss-120b-GGUF (I'm using the Q4_K_M variant) and run:

llama-server -m "C:\<MODEL_PATH>\gpt-oss-120b-Q4_K_M-00001-of-00002.gguf" -ngl 99 --jinja

This is the output that I see:
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG...

I can confirm this happens deterministically. I can also confirm the problem does not happen when the --jinja flag is removed.

[ggerganov]

Does it work if you add -fa?

[danielholanda]

No, unfortunately enabling Flash Attention does not help. I see the same exact GGGG issue.

llama-server -m "C:\<MODEL_PATH>\gpt-oss-120b-Q4_K_M-00001-of-00002.gguf" -ngl 99 --jinja -fa

[tylerstraub]

looks like potentially the same issue reported back in May but auto closed due to inactivity: #13310

could it be a backend/Jinja interaction rather than a model-specific issue?

[Dillflix]

I'm hitting a similar issue with gpt-oss-120b + AMD RyzenAI MAX+ 395 (gfx1151), with both the HIP and Vulkan backends.

For me it occurs:

• With both ROCm 6.4.2 and ROCm 7.0rc.
• With HIP and Vulkan backends.
• With and without flash attention.
• With varying -b and -ub sizes from 512-8192.
• With all llama.cpp versions/commits I've attempted to use.
• With GGUFs from different sources and various quants (mxfp4, FP16, BF16, etc).
• On Linux; haven't tested on Windows.

The only scenario where the issue has not occurred is when adding a discrete GPU (7900XT, gfx1100) and offloading only the expert tensors to the gfx1151, ie:

• All ffn + attn tensors on the gfx1100
• All expert tensors on the gfx1151
  With this setup it seems to work as intended, or at least works in the same scenarios that fail when using only the gfx1151.

So I think this issue is going be hardware + model specific, for AMD Strix Halo (gfx1151) + gpt-oss.

I'll gladly provide any logging that could help debug it and/or test any relevant changes.

[Dillflix]

However, for me it seems to be occurring with and without --jinja.

[Mushoz]

I am seeing similar issues with gfx1151 with the Vulkan backend (haven't tried HIP). Strangely, it doesn't always happen with every prompts. Most of them are fine. But if a prompt triggers the issue, rerunning the same prompt triggers the same issue yet again. I have seen scenario's where the G character is spammed indefinitely (GGGGGGGGGGGG), but I have also seen the model spam : dissolution dissolution dissolution dissolution dissolution

Happy to help debug if there is anything I can do.

[Mushoz]

By the way, all other models I have tried are fine. So it's really a gpt-oss specific issue. Could this be triggered by the attention sinks by any chance?

[jdilly]

Strangely, it doesn't always happen with every prompts. Most of them are fine. But if a prompt triggers the issue, rerunning the same prompt triggers the same issue yet again. I have seen scenario's where the G character is spammed indefinitely (GGGGGGGGGGGG), but I have also seen the model spam : dissolution dissolution dissolution dissolution dissolution

It's almost always "dissolution" being spammed repeatedly for me, with both HIP and Vulkan, but otherwise identical to this.

[Mushoz]

Strangely, it doesn't always happen with every prompts. Most of them are fine. But if a prompt triggers the issue, rerunning the same prompt triggers the same issue yet again. I have seen scenario's where the G character is spammed indefinitely (GGGGGGGGGGGG), but I have also seen the model spam : dissolution dissolution dissolution dissolution dissolution

It's almost always "dissolution" being spammed repeatedly for me, with both HIP and Vulkan, but otherwise identical to this.

Dissolution spam is by far the most common for me as well.

[ggerganov]

The only scenario where the issue has not occurred is when adding a discrete GPU (7900XT, gfx1100) and offloading only the expert tensors to the gfx1151, ie:

• All ffn + attn tensors on the gfx1100
• All expert tensors on the gfx1151

With this setup it seems to work as intended, or at least works in the same scenarios that fail when using only the gfx1151.
So I think this issue is going be hardware + model specific, for AMD Strix Halo (gfx1151) + gpt-oss.

Seems like something goes wrong in the attention. Since it works on gfx1100 and not on the gfx1151 it makes me think it could be a driver problem? Not sure (pinging @0cc4m for any ideas).

@Dillflix Could you confirm the following:

• With gfx1100 and -fa -ngl 99 --cpu-moe it works
• With gfx1151 and -fa -ngl 99 --cpu-moe it does not work

The idea of the test is to keep the experts on the CPU and run the attention on the different cards under the same conditions.

[Mushoz]

it makes me think it could be a driver problem

Well the OP is on Windows with Windows drivers, while I tested radv, pro and amdvlk under Linux with the same results.

I guess it could still be in the linux drivers, since the OP reported a spam of GGGGGGG (which I think I have only personally seen once), while me and @jdilly see the Dissolution spam, so then the Dissolution spam could be a linux specific issue?

[ceroma]

Just to add another data point, I'm hitting the same issue with the following config:

• latest version of the model from ggml-org/gpt-oss-120b-GGUF
• llama.cpp b049315 built with Vulkan support (-DGGML_VULKAN=1)
• Arch Linux + Ryzen AI HX 370 (Radeon 890M gfx1150 Strix Point) + 128GB DDR5 RAM
• Base run command: llama-server --model models/gpt-oss-120b-mxfp4-00001-of-00003.gguf --n-gpu-layers 99 --ctx-size 0 --flash-attn --jinja

Also tried:

• unsloth/gpt-oss-120b-GGUF's FP16
• different values for --ctx-size
• different values for --batch-size
• with and without --jinja
• with and without --flash-attn
• with and without --reasoning-format
• with and without --cpu-moe

When I hit the issue, I'm always getting dissolution now but I remember seeing GGGG in my initial tests when I mistakenly used Qwen3's recommended temp/top-k/top-p params instead.

In all my tests, I usually get a coherent response for my first message, but if the initial response exceeds ~1-2k tokens, then the next response I get is just dissolution dissolution ....

[ggerganov]

@ceroma Could you add -ctk bf16 -ctv bf16 and see if it still occurs?