RDNA4 (gfx1201 / Radeon AI PRO R9700): prebuilt `llama-server` segfaults on startup — plus an iGPU/HIP crash Studio doesn't guard against

[AiwendilInTheWoods]

Edited — corrections in-line, nothing deleted. Problem 2's diagnosis was wrong: the cause
was the integrated GPU being visible to HIP, not AllReduce or context size. The unnumbered
--tensor-parallel note at the end was also wrong. Both are struck through below.
Problems 1 and 3–8 are unchanged.

Hey folks — first off, congratulations on the AMD launch, and thank you for it. Getting Unsloth
onto Radeon is a big deal for those of us who went AMD for the VRAM-per-dollar and then spent a
year watching the good tooling land on CUDA first. We were genuinely excited to try this the day
it dropped.

We have a slightly unusual box — two Radeon AI PRO R9700s (gfx1201) — which we think sits
outside your current AMD CI, so hopefully this is useful rather than annoying. We hit a wall, but
we did a fair bit of digging to narrow it down, and we're very happy to keep digging or test
patches if that helps. Short version: the published RDNA4 prebuilt won't start at all, and
when we built your fork from source to work around it, we found a second, separate crash in what
looks like the multi-GPU path.

We've tried to make everything below reproducible. Happy to run anything you'd like.

Our setup

GPUs	2× AMD Radeon AI PRO R9700 (gfx1201, 32 GB each) + a Ryzen iGPU (excluded via HIP_VISIBLE_DEVICES) (correction: the iGPU was visible in the Problem 2 runs — that was the cause)
CPU	AMD Ryzen 9 9950X (16C/32T)
RAM	64 GB
OS	Ubuntu 24.04.4 LTS, kernel 6.8.0-124-generic
Host ROCm	7.2.3
amdgpu-dkms	1:6.16.13.30300300
Install	curl -fsSL https://unsloth.ai/install.sh | sh, as an unprivileged user
unsloth	2026.7.4
torch	2.10.0+rocm7.2.3 (from repo.radeon.com/rocm/manylinux/rocm-rel-7.2.3/)
llama.cpp asset	app-b10069-mix-fb3d4ca-linux-x64-rocm-gfx120X.tar.gz
release	unslothai/llama.cpp@b10069-mix-fb3d4ca (tag b10069)
archive sha256	e82239ad779674fd0fedffe0222195fc0b5b27e2cd7a1ef83b2455d7292425c7 — the installer verified this, so it isn't a corrupted download

First, the things that went really well

We want to lead with this, because the install experience was mostly excellent and it helps
localise the problem:

• gfx1201 was detected correctly even with two discrete GPUs and an iGPU in the box. We'd
  half-expected that to be the hard part, and it just worked.
• PyTorch resolved to our exact host ROCm — torch 2.10.0+rocm7.2.3 straight from the Radeon
  repo. That's better than the docs led us to expect (they suggest nothing exists above ROCm 7.2).
• The vision projector was auto-detected sitting next to the model, with zero configuration.
  Lovely touch.
• GGUF metadata parsing was spot on — context length, chat template, reasoning and tool-calling
  flags all read correctly.
• The prebuilt bundles its own ROCm 7.14 runtime, which nicely decouples it from whatever the
  host happens to have. Smart.
• --disable-tools and the Cloudflare-off default both behaved exactly as documented, and the API
  correctly returned 401 without a bearer token. We care about that a lot and it was right.

So this really does look like a problem with one published artifact rather than anything
fundamental.

Problem 1: the gfx120X prebuilt segfaults immediately

The binary the installer places doesn't run at all — not even --version:

$ cd ~/.unsloth/llama.cpp
$ export LD_LIBRARY_PATH=$HOME/.unsloth/llama.cpp/build/bin
$ ./llama-server --version
Segmentation fault (core dumped)     # exit 139

From the kernel log:

llama-server[...]: segfault at 100000001 ip 0000000100000001 sp 00007fff1ecd7b78
                   error 14 in llama-server[5d94eac9d000+1000] likely on CPU 0

The instruction pointer is 0x100000001 and equals the faulting address, which reads like a jump
through a corrupt or uninitialised function pointer very early in startup — before any model,
ROCm, or GPU work happens. A few things support that reading:

• it fails the same way on --version and --help, which touch no hardware at all
• VRAM never moves off idle (~57 MiB/card) on any attempt
• the fault address is identical across runs; only the load base changes

Studio's fallback chain retried five times and every attempt died identically:

attempt	flags	result
1	--fit off --flash-attn on --split-mode layer + --mmproj	exit -11
2	--fit on retry	exit -11
3	--flash-attn off	exit -11
4	--fit on --flash-attn off	exit -11
5	text-only, no --mmproj	exit -11

One small thing that made debugging harder: the bundle ships only llama-server — there's no
llama-cli, llama-bench or llama-mtmd-cli to cross-check against. Including those would have
saved us a while.

To reproduce: install on any gfx1201 host and run ~/.unsloth/llama.cpp/llama-server --version.

Problem 2: building your fork from source fixes that — but reveals a different crash [diagnosis corrected — the cause was the iGPU]

To work out whether this was packaging or code, we built the fork ourselves on the same machine:

git clone https://github.com/unslothai/llama.cpp.git
HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
  cmake -S . -B build -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1201 \
        -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=ON
cmake --build build --config Release -j 24

HEAD was 5ff665dc (git describe → b10068-mix-fb3d4ca); the binary self-reports
version: 9690 (5ff665dc). Small note — we couldn't pin the prebuilt's exact ref because
fb3d4ca turns out to be part of the tag name rather than a commit hash, which took us a moment
to work out.

The good news: the source build runs fine (--version exits 0) and it has MTP —
--spec-type ... draft-mtp and --spec-draft-n-max are both there. So Problem 1, and the missing
MTP in Problem 3 below, both look like packaging or build-configuration issues rather than
anything wrong with your code. That felt like the most useful thing we could tell you.

⚠️ Retracted. The crash is real; the diagnosis was wrong. Cause: the Ryzen iGPU visible to
HIP. Context size, --mmproj, -sm tensor, MTP and AllReduce are all irrelevant. With
HIP_VISIBLE_DEVICES set to the two discrete cards this command loads and serves.

The less good news: the source build still segfaults on our actual production configuration:

./llama-server --host 0.0.0.0 --port 9310 \
  -m Qwen3.6-27B-UD-Q6_K_XL.gguf --mmproj mmproj-F16.gguf \
  -c 262144 -sm tensor -ngl 999 -fa 1 -np 1 \
  --spec-type draft-mtp --spec-draft-n-max 3 --fit off --kv-unified -ctk q8_0 -ctv q8_0

-> Segmentation fault (core dumped); VRAM never allocated (~57 MiB/card idle)

The last line logged before the crash, every single time:

W internal AllReduce init failed (n_devices != 2?); falling back to meta-backend butterfly

We bisected the flags at -c 4096 and everything loads happily — baseline, -sm tensor,
--spec-type draft-mtp, --mmproj, and tensor+MTP together — with zero AllReduce warnings.
So it seems to need large context (and in our runs, the vision projector) before it shows up.

Here's the bit we think is most actionable. We run the same model, same flags, same two GPUs
on an upstream-derived llama.cpp build 9860, and it prints that same
AllReduce init failed (n_devices != 2?) warning — and then carries on and serves perfectly
happily at 262144 context with the vision projector loaded, using ~38.9 GiB across the two cards.

✅ Stands. The warning appears on runs that load and serve normally, so it is not a crash
indicator on either build.

On your fork's build 9690, the same warning is followed by a segfault. So something in that
multi-GPU AllReduce fallback either regressed, or is being reached in a state the fallback path
can't cope with.

We're very happy to run a debug build, grab a backtrace, or bisect further if that would help —
just say the word.

Problem 3: the gfx120X prebuilt has no MTP support

On startup Studio logs:

llama.cpp prebuilt lacks MTP support (--spec-type mtp/draft-mtp).
Run `unsloth studio update`. MTP GGUFs will load without speculative decoding.

Since the source build does have MTP, this looks like the same packaging story as Problem 1.
Worth flagging because it matters a lot on this hardware: on our working build, MTP is worth
1.63–1.79× generation throughput on code and reasoning workloads, so quietly dropping it is a
big hit — especially for the Qwen3.6 and Gemma 4 families the launch post highlights.

Problem 4: unsloth studio update can't resolve it

The remedy in that warning is a no-op, because b10069 already is the latest published release:

existing install detected -- validating update
llama.cpp      prebuilt up to date and validated
               installed release: unslothai/llama.cpp@b10069-mix-fb3d4ca (tag b10069)

So as far as we can tell there's currently no supported route to a working RDNA4 install.

Problem 5: the update path reports the CPU as the GPU

Minor, but confusing while debugging. The install path correctly says
gpu AMD ROCm (gfx1201). The update path, same machine, same session, says:

GPU: AMD Ryzen 9 9950X 16-Core Processor

Looks like the two paths use different detection logic.

Problem 6: the crash gets misattributed

This one cost us real time, so it might be worth a look. The segfault surfaces as two different
and both incorrect diagnoses:

llama-server crashed during startup (exit code -11) with forced --fit off;
the fit estimate was optimistic, retrying once with --fit on so it can offload.

llama-server could not load this model's vision projector (--mmproj).
The installed llama.cpp build is likely too old for it. Loading text-only for this session

with a final user-facing error of "Vision projector incompatible with this llama.cpp build, and
the text-only retry also failed" — when really the binary never started. We went looking at our
mmproj file and our VRAM headroom for a while before checking dmesg. Since a negative exit code
(-11 = SIGSEGV) is unambiguous, surfacing it as "the child crashed with SIGSEGV" would point
people at the real problem much faster.

Problem 7: installing without sudo, non-interactively

Installing as a user with no sudo and no TTY:

deps           missing: libcurl4-openssl-dev
    WARNING: We require sudo elevated permissions to install: libcurl4-openssl-dev
    Accept? [Y/n] sh: 593: cannot open /dev/tty: No such device or address
sudo: a terminal is required to read the password
curl: (23) Failure writing output to destination
ERROR: Process exited with status 1

Two small things bundled together: with no TTY the prompt hard-fails rather than detecting
non-interactive mode and printing what to install; and the build dependencies are required up
front, before checking whether a prebuilt exists — even though the whole point of the prebuilts
(per the AMD post) is avoiding local compilation. For CI, containers, or locked-down accounts,
that's a stop for tooling that may never get used.

Problem 8: the startup reachability probe

On startup, Studio reached out to external probe nodes and had them attempt an inbound
connection to the bound port, then logged the machine's public egress IP:

Reachability check: http://<PUBLIC-IP-REDACTED>:8090/ is NOT reachable from the public
internet (2/2 probe nodes failed).

We understand what it's for and it's a genuinely nice feature for someone trying to share a
Studio. But it's undocumented, happens without asking, and discloses the deployment's public IP
plus the presence of a listening service to a third party. For lab, air-gapped, or
privacy-sensitive deployments that's a surprise. Could it be opt-in, or at least have an env var
to switch it off?

One more small thing — retracted

We passed unsloth run --tensor-parallel and saw --split-mode layer appended to the
llama-server command line rather than tensor split. That may just be the fallback chain degrading
after the first crash, so treat it as low confidence until the segfault is sorted.

❌ Retracted — was my error. --tensor-parallel correctly produces --split-mode tensor.
The original reading came from the last line of Studio's multi-attempt fallback chain rather
than the first; the last line is a degraded retry. Studio is behaving correctly here.

Happy to help

We've got dual R9700 (gfx1201) hardware, a fully instrumented baseline of the same models on a
working llama.cpp build, and time to spend on this. If a fix or a test build would benefit from
someone running it on real RDNA4 multi-GPU, we're glad to be that someone — debug builds,
backtraces, bisects, before/after numbers, whatever's useful.

Thanks again for the AMD work. Really looking forward to running this properly.
(We've bundled several things into one issue to keep the story together — the packaging problem,
the source-build crash we found while proving it was packaging, and a few smaller things we noticed
along the way. Very happy to split any of it into separate issues if that's easier for you to
track.)

[AiwendilInTheWoods]

Correction + a much better root cause for Problem 2 — it's the integrated GPU, not context size.

Apologies, we mischaracterised Problem 2 in the original post. Someone asked us the obvious question
("does this happen with one GPU?"), we went to test it properly, and the answer turned out to be a
lot more useful than what we'd originally written. Correcting the record, because the real cause is
both simpler and affects far more people than we implied.

What we got wrong

We said the source build "segfaults reproducibly on a large-context multimodal config" and pointed at
the AllReduce init failed (n_devices != 2?) warning as the signal. Both of those were wrong.

• The AllReduce warning is a red herring. It appears on every 2-GPU run on this box, including
  all the ones that load and serve perfectly. It just happens to be the last line printed before the
  crash. Our mistake.
• Context size, --mmproj, -sm tensor and MTP are all irrelevant — we've now loaded 262144
  context with the vision projector, tensor split and MTP many times without incident.

The actual cause: the iGPU being visible to HIP

This machine has two discrete R9700s plus the Ryzen 9950X's integrated GPU. Same binary, same
flags, same model, only difference is whether the iGPU is hidden:

# crashes, every time
./llama-server --host 0.0.0.0 --port 9601 -m Qwen3.6-27B-UD-Q6_K_XL.gguf \
  --mmproj mmproj-F16.gguf -c 262144 -sm tensor -ngl 999 -fa 1 -np 1 \
  --spec-type draft-mtp --spec-draft-n-max 3 --fit off --kv-unified -ctk q8_0 -ctv q8_0
-> Segmentation fault (exit 139)

# identical command, loads and serves fine
HIP_VISIBLE_DEVICES=0,1 ./llama-server ...same flags...
-> LOADED

Kernel log for the crashing case — note it's a GP fault inside the HIP runtime, not in
llama-server:

traps: llama-server[2001676] general protection fault ip:72cffdab097f sp:7ffce7a3c2e0
       error:0 in libamdhip64.so.7.2.70203[72cffd823000+481000]

That's a different signature from Problem 1 (segfault at 100000001 ip 0000000100000001, inside
llama-server itself), which is how we finally noticed these are two unrelated bugs.

This also looks like the same class of problem reported elsewhere for R9700 + iGPU
(e.g. lemonade-sdk/llamacpp-rocm issue #96), so it may not be yours to fix in llama.cpp at all.

Why we think it still matters for Unsloth

Unsloth Studio doesn't set HIP_VISIBLE_DEVICES when it launches llama-server. We can see the
full command line it builds in the logs, and there's no device filtering in it. That means on any
machine where a discrete Radeon sits alongside an integrated GPU, Studio hands the HIP runtime a
device list that includes the iGPU — and on our hardware that's an immediate crash.

That's a much larger population than our slightly odd dual-GPU box: essentially every Ryzen desktop
or workstation with integrated graphics plus a discrete Radeon. Some options that would have saved us
(and probably others):

• default to discrete devices only, or expose a device-selection setting in Studio
• set HIP_VISIBLE_DEVICES explicitly from whatever Studio's own detection already worked out — it
  correctly identified gfx1201 at install time, so the information is clearly available
• or simply document it, if filtering automatically is the wrong call

What still stands unchanged

Problem 1 is unaffected by any of this. We re-tested the shipped prebuilt with the iGPU hidden and
it still segfaults on --version:

./llama-server --version                              -> exit 139
HIP_VISIBLE_DEVICES=0,1 ./llama-server --version      -> exit 139
HIP_VISIBLE_DEVICES=0,1 ./llama-server -m <model> ... -> exit 139

So the prebuilt is genuinely broken independently of the iGPU issue, and RDNA4 users are still
blocked on it regardless of their hardware layout. Problems 3–8 are unchanged too.

Net

Two separate bugs, and the second one has a trivial workaround we didn't know about when we filed:
set HIP_VISIBLE_DEVICES to your discrete cards. Sorry for the noise on the original diagnosis —
hopefully the corrected version is more useful. Happy to test any device-filtering change on this
box, iGPU and all.

---

Edited addendum:

1. The crash-signature difference above is supporting detail, not proof. The proof is the
   re-test: the prebuilt still segfaults on --version with the iGPU hidden.
2. "Problems 3–8 are unchanged" did not cover the unnumbered "One more small thing" about
   --tensor-parallel. That was also wrong — now struck through in the body.

[danielhanchen]

Thanks for the thorough writeup. Two separate things here:

1. The prebuilt llama-server startup segfault should be fixed by fix(rocm): prepend system ROCm libs on native Linux to avoid bundled HIP crash #7233 (merged to main). On native Linux it now prepends the system ROCm libs ahead of the bundled HIP runtime; that logic is arch-agnostic, so it applies to gfx1201 as well. Once the next release ships (or if you build from main), could you retest whether llama-server starts on the R9700s?

2. The multi-GPU AllReduce crash in the source build looks like a separate llama.cpp issue and is not covered by that change. If it still reproduces after (1), a backtrace from the source build would help us narrow it down.

[AiwendilInTheWoods]

Retested — fix confirmed on gfx1201. The mechanism is library ordering, so it didn't need the release:

LD_LIBRARY_PATH	llama-server --version
bundled first (2026.7.4 today)	exit 139
/opt/rocm/lib prepended	exit 0 — version: 10069 (72051453a)

With system ROCm prepended the prebuilt loads for real: Qwen3.6-27B Q6_K, -sm tensor, both R9700s, 24.7 GiB, serving completions at ~37 t/s.

Matches the PR: installed 2026.7.4 has only _wsl_system_rocm_lib_dirs() gated on /dev/dxg, so native Linux never got the prepend. Caveat — that exercises the mechanism, not the shipped code path; happy to retest the release build.

On point 2: no backtrace to give — that diagnosis is retracted, see the strikethroughs in the body. The crash was the iGPU visible to HIP.