Studio: add Vulkan llama.cpp support#5819
Conversation
for more information, see https://pre-commit.ci
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for the Vulkan backend in llama.cpp, enabling GPU acceleration on Intel and other non-NVIDIA/non-AMD GPUs. It adds a short-lived subprocess probe script to query Vulkan VRAM without initializing Vulkan in the main process, handles device pinning via GGML_VK_VISIBLE_DEVICES, and updates the prebuilt installer to detect Intel GPUs and fetch the appropriate Vulkan assets. The review feedback highlights two key improvements: resolving a potential AttributeError on Windows where ctypes.RTLD_GLOBAL is undefined, and checking the subprocess return code and logging stderr to prevent silent failures during the Vulkan GPU probe.
for more information, see https://pre-commit.ci
|
Tested install + inference on my dual NVIDIA GPU setup by overriding the NVIDIA detection and forcing vulkan. Ended up discovering a bug that I fixed at 4fefeeb. After that, the test worked perfectly end to end, with a small model pinning successfully to only one of the GPUs and generating text. |
for more information, see https://pre-commit.ci
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request adds support for the Vulkan backend in llama.cpp, enabling GPU acceleration on Intel and other non-NVIDIA/non-AMD GPUs. It introduces a standalone _vulkan_probe.py script to query free VRAM, integrates Vulkan memory detection and device pinning in llama_cpp.py, adds regression tests, and updates the prebuilt installer to detect Intel GPUs and support Vulkan prebuilt installations. A critical issue was identified in _vulkan_probe.py where accessing ctypes.RTLD_GLOBAL on Windows will raise an AttributeError and crash the probe subprocess; a suggestion was provided to conditionally apply RTLD_GLOBAL only on non-Windows platforms.
# Conflicts: # studio/backend/core/inference/llama_cpp.py # studio/install_llama_prebuilt.py
for more information, see https://pre-commit.ci
# Conflicts: # studio/backend/core/inference/llama_cpp.py # studio/install_llama_prebuilt.py
for more information, see https://pre-commit.ci
# Conflicts: # studio/backend/core/inference/llama_cpp.py
# Conflicts: # studio/backend/core/inference/llama_cpp.py
|
@codex review |
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5432a0c168
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| the torch/training stack is installed separately and still sees the real | ||
| GPU. | ||
| """ | ||
| return os.environ.get("UNSLOTH_FORCE_VULKAN", "").strip().lower() in ( |
There was a problem hiding this comment.
When UNSLOTH_FORCE_VULKAN is used on an AMD/NVIDIA host, this env check only affects the setup-time installer invocation. The in-app updater later runs install_llama_prebuilt.py from utils/llama_cpp_update.py with the marker repo/asset and ROCm args, but it does not recreate this env flag; if the user only set it for setup, the next update routes by detected ROCm/CUDA instead of preserving the installed Vulkan choice (or can fail on Linux ROCm against upstream). Please derive the force from the existing Vulkan asset/marker or persist it in metadata so updates keep the Vulkan prebuilt.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in f455558. The update apply step now re-asserts UNSLOTH_FORCE_VULKAN in the installer subprocess env when the marker asset is a Vulkan bundle, mirroring how _rocm_install_args forwards ROCm from the asset name. Without it, detect_host on a CUDA/ROCm box would re-route the update to that backend and silently replace the Vulkan build. Test: test_start_update_preserves_vulkan_via_env.
| if not LlamaCppBackend._is_vulkan_backend(binary) and LlamaCppBackend._get_gpu_free_memory( | ||
| binary | ||
| ): |
There was a problem hiding this comment.
On CUDA/ROCm machines that install the Vulkan llama-server via UNSLOTH_FORCE_VULKAN, this guard makes RAG auto skip the existing sentence-transformers path solely because the llama.cpp binary is Vulkan. Those hosts can still have torch-visible CUDA/ROCm GPUs (the force flag is scoped to llama.cpp), so bulk indexing regresses to the llama-server embedder even though the function's contract is to use ST when a CUDA/ROCm GPU is present. Check torch/CUDA/ROCm availability independently of the llama-server backend before treating Vulkan as non-torch-usable.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Good catch, and you're right that the guard I added last round regressed a force-Vulkan-on-CUDA host. Rather than add torch-independent GPU detection to _resolve_auto (which drags RAG backend selection into this Vulkan-install PR), I reverted that guard in f455558 so _resolve_auto is back to its original pre-PR behavior. The torch-vs-llama.cpp-backend conflation in RAG auto-selection is a pre-existing concern orthogonal to adding Vulkan install support and is better handled on its own.
| binary = binary or LlamaCppBackend._find_llama_server_binary() | ||
| if not binary: | ||
| return False | ||
| return (_llama_lib_dir(binary) / _vulkan_lib_filename()).is_file() |
There was a problem hiding this comment.
When Studio is pointed at a local/custom llama.cpp build that includes multiple ggml backends, the presence of libggml-vulkan now makes the whole runtime behave as Vulkan-only. That bypasses the existing NVIDIA/ROCm memory probes and later emits Vulkan device names, so a build that also has CUDA or HIP can be sized and launched against the wrong backend (or report no GPU if the Vulkan probe cannot enumerate devices). Please detect the selected install kind or prefer CUDA/HIP backend libraries before treating this sibling file as exclusive.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in f455558. _is_vulkan_backend now returns True only when Vulkan is the sole GPU backend: if a libggml-cuda or libggml-hip sibling sits next to libggml-vulkan (a custom multi-backend build), it defers to the CUDA/HIP path (nvidia-smi/torch probes and the CUDA/HIP pin). Official prebuilts are single-backend so they're unaffected. Tests: test_multi_backend_build_is_not_vulkan_only and test_vulkan_only_build_is_detected.
…nd preserve forced Vulkan across updates
|
@codex review |
1 similar comment
|
@codex review |
for more information, see https://pre-commit.ci
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4e968d483a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| base = ctypes.CDLL(os.path.join(bindir, base_name), mode = ctypes.RTLD_GLOBAL) | ||
| lib = ctypes.CDLL(os.path.join(bindir, vk_name), mode = ctypes.RTLD_GLOBAL) |
There was a problem hiding this comment.
On Windows ctypes does not define RTLD_GLOBAL, so the attribute lookup here raises AttributeError before ctypes.CDLL runs, and the except OSError block does not catch it. For every Windows Vulkan install the probe subprocess exits non-zero and _get_gpu_free_memory_vulkan() returns [], which removes Vulkan VRAM sizing and device pinning even though the bundle is installed; use getattr(ctypes, "RTLD_GLOBAL", 0) or omit mode on Windows.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Addressed in 40c4686 with getattr(ctypes, 'RTLD_GLOBAL', 0). To be accurate on the mechanism: ctypes.RTLD_GLOBAL is defined on every platform, including Windows, where CPython's _ctypes falls back to 0 (#ifdef RTLD_GLOBAL #else 0), and CDLL ignores mode on Windows and uses LoadLibraryEx, so there was no AttributeError and the probe already loaded. The getattr form is behaviorally identical and makes the cross-platform default explicit, so I've applied it to close this out.
|
@codex review |
|
Codex Review: Didn't find any major issues. Bravo. Reviewed commit: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
On a Vulkan llama.cpp build gpu_indices are ggml compact ordinals, not CUDA/ROCm physical ids, so _amd_apu_wants_unified_memory and _apply_datacenter_env were reading the wrong device. On a mixed AMD APU plus discrete GPU host that could raise a spurious system-RAM shortfall and block a valid discrete-GPU load. Gate all three call sites on not is_vulkan_backend; the Vulkan path already reserves iGPU host headroom and the backend ignores GGML_CUDA_* anyway.
|
Pushed a small fix on top ( Three call sites in
The env tunings are harmless on Vulkan since the backend ignores Fix gates all three on |
for more information, see https://pre-commit.ci
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d1b37e875d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| base.ggml_backend_reg_dev_count.restype = ctypes.c_size_t | ||
| base.ggml_backend_reg_dev_count.argtypes = [ctypes.c_void_p] | ||
| base.ggml_backend_reg_dev_get.restype = ctypes.c_void_p | ||
| base.ggml_backend_reg_dev_get.argtypes = [ctypes.c_void_p, ctypes.c_size_t] | ||
| base.ggml_backend_dev_type.restype = ctypes.c_int | ||
| base.ggml_backend_dev_type.argtypes = [ctypes.c_void_p] |
There was a problem hiding this comment.
In the upstream shared-library layout, the registry APIs (ggml_backend_reg_dev_count / ggml_backend_reg_dev_get) live in libggml.so, not libggml-base.so; looking them up on the base handle here makes _igpu_flags() catch an AttributeError and return all-False. On Vulkan integrated-GPU installs this means the parent never applies the iGPU host-memory reserve and passes the shared-memory total through as if it were discrete VRAM, so context fitting can over-budget host RAM. Load/use the ggml registry library for these calls instead of libggml-base.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Not a bug. nm -D on the actual upstream Vulkan prebuilt (b9935) shows ggml_backend_reg_dev_count, ggml_backend_reg_dev_get and ggml_backend_dev_type are all defined and exported (T) in libggml-base.so; in libggml.so they are only imported (undefined U), and libggml-vulkan.so lists libggml-base.so.0 in DT_NEEDED. So loading libggml-base and looking these up on that handle resolves cleanly (verified via ctypes), _igpu_flags reads the device type, and the iGPU host-reserve / total=0 path works. Left as is.
| def _llama_lib_dir(binary: str) -> Path: | ||
| # The installer exposes llama-server as a top-level symlink into build/bin/, | ||
| # where the ggml backend libs live. Resolve it so callers looking for | ||
| # sibling libs (Vulkan detection, LD_LIBRARY_PATH, probe bindir) hit the | ||
| # real directory, not the symlink's parent. | ||
| return Path(binary).resolve().parent |
There was a problem hiding this comment.
When create_exec_entrypoint() cannot create the top-level symlink it writes a shell wrapper at install_dir/llama-server, and _find_llama_server_binary() still prioritizes that root entrypoint. Path.resolve() does not follow the wrapper to build/bin, so on those supported installs this returns install_dir; _is_vulkan_backend() then misses build/bin/libggml-vulkan.so, skipping the Vulkan probe and --device pinning for an otherwise valid Vulkan install. Handle the wrapper fallback (or prefer the real build/bin binary) here as well as symlinks.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 576d81c. create_exec_entrypoint's fallback writes install_dir/llama-server as a #!/bin/sh exec-wrapper when it cannot symlink; Path.resolve() does not follow that wrapper, so _llama_lib_dir returned the install root and _is_vulkan_backend missed build/bin/libggml-vulkan.so, silently skipping the Vulkan probe and --device pin. _llama_lib_dir now follows the wrapper's exec target to build/bin. Regression test: test_shell_wrapper_entrypoint_resolves_to_real_lib_dir.
create_exec_entrypoint falls back to a #!/bin/sh wrapper at the install root when it cannot symlink into build/bin. _find_llama_server_binary returns that root entrypoint, but Path.resolve() does not follow a shell wrapper, so _llama_lib_dir returned the install root and _is_vulkan_backend missed libggml-vulkan.so -- silently skipping the Vulkan probe and --device pin on an otherwise valid Vulkan install. Follow the wrapper's exec target to build/bin. Regression test: test_shell_wrapper_entrypoint_resolves_to_real_lib_dir.
for more information, see https://pre-commit.ci
|
@codex review |
|
Codex Review: Didn't find any major issues. More of your lovely PRs please. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Cloned latest git version, installed unsloth with UNSLOTH_FORCE_VULKAN=1 but it still running on ROCM backend. |
Bring the Intel XPU Studio enablement up to date with main (post unslothai#5819) and resolve the four conflicts additively, keeping every non-XPU backend (NVIDIA CUDA, AMD ROCm, CPU, Apple MLX) byte-for-byte unchanged. - utils/hardware/hardware.py: re-apply the XPU points on top of main's refactor -- detect_hardware explicit-XPU hint, XPU as a supported selection backend (auto_select_gpu_ids, prepare_gpu_selection, get_device_map), ZE_AFFINITY_MASK visibility (_get_parent_visible_gpu_spec, get_visible_gpu_count, _backend_visible_devices_env, apply_gpu_ids), backend-aware clear_gpu_cache, package versions, per-device info, and the fork-after-Level-Zero-init guard. CUDA/ROCm/CPU/MLX paths are untouched. - core/training/trainer.py: route the three codec cache clears through the backend-aware clear_gpu_cache(); keep codec device as CUDA-or-CPU because the SNAC/Spark-TTS/OuteTTS codecs are not yet validated on XPU. - core/inference/llama_cpp.py: take main's Vulkan GGUF design verbatim; llama.cpp Intel inference (Vulkan) is decoupled from torch XPU training. - tests/test_gpu_selection.py: XPU is now a supported selection backend.
4 participants
Summary
This adds Vulkan support to Studio so Intel GPUs get GPU-accelerated llama.cpp inference instead of falling back to CPU. Intel GPUs use Vulkan, not CUDA or ROCm, so today Studio treats them as having no GPU at all. The change reuses Studio's existing VRAM and GPU-selection code, so once we know how much free VRAM there is, context auto-sizing, multi-GPU selection, and layer offload all work the same way they do for NVIDIA and AMD.
How it works
Reading free VRAM through llama.cpp's own Vulkan library
Instead of parsing the text output of a command-line tool, it loads the Vulkan library that already ships with llama.cpp (
libggml-vulkan.soon Linux,ggml-vulkan.dllon Windows) and callsggml_backend_vk_get_device_countandggml_backend_vk_get_device_memorydirectly to get free and total VRAM per GPU. A side effect is that the GPU indices come back in llama.cpp's own order, which is what the GPU-pinning step below needs. The library is loaded in a short-lived helper process so no Vulkan context lives inside the main Studio process._get_gpu_free_memory()now dispatches by backend: on a Vulkan build it uses the Vulkan reader; otherwise it runs the existing NVIDIA (nvidia-smi/torch) and AMD ROCm path. The NVIDIA/ROCm implementation is unchanged, just moved into a renamed helper.Leaving host headroom on integrated GPUs
On an integrated GPU, ggml reports shared system RAM as "VRAM" (it sums every memory heap), so auto-sizing context and offload against that number could claim nearly all of RAM and push the host into swap or the OOM killer. For integrated GPUs the reader leaves a per-device host margin of a flat 1 GiB, matching llama.cpp's own
--fit-targetdefault, instead of inventing a larger reserve. Whether a device is integrated comes straight from ggml's device type (GGML_BACKEND_DEVICE_TYPE_IGPU), read in the same helper process, so discrete cards are never touched (no VRAM-vs-RAM ratio guessing).Pinning the GPU with
GGML_VK_VISIBLE_DEVICESWhen a model fits, Studio already picks the GPU(s) and puts all layers on them (
-ngl -1). For CUDA and ROCm it does this withCUDA_VISIBLE_DEVICES/HIP_VISIBLE_DEVICES/ROCR_VISIBLE_DEVICES. The Vulkan backend ignores those, so on a Vulkan build we use its own setting,GGML_VK_VISIBLE_DEVICES. The indices come from the same library that read the VRAM, so the pin always targets the right device. Multi-GPU works the same way it does for CUDA and ROCm.Picking the Vulkan build at install time
The Vulkan builds come from upstream ggml-org/llama.cpp at the same tag, the same way the Windows-CUDA, CPU, and macOS builds already do, so there is nothing new to build or publish (
llama-<tag>-bin-ubuntu-vulkan-x64.tar.gzandllama-<tag>-bin-win-vulkan-x64.zip).The installer picks the Vulkan build only when there is no NVIDIA GPU, no ROCm GPU, and an Intel GPU is detected. Intel detection is:
/sys/class/drm/card*/device/vendorand checks for0x8086(Intel).Get-CimInstance Win32_VideoControllerand looks for "Intel".The new
linux-vulkanandwindows-vulkaninstall kinds are wired through the same places the others already go through (runtime file patterns, GPU validation, install health check).Letting AMD users opt into Vulkan
AMD GPUs run Vulkan too, but by default Studio installs the ROCm build for them. Setting
UNSLOTH_FORCE_VULKAN=1before setup installs the upstream Vulkan build instead. This helps where ROCm is awkward (e.g. some integrated parts), and the integrated-GPU host margin above keeps it safe on shared-memory hardware like Strix Halo. It's scoped to the llama.cpp inference backend. The torch/training stack is a separate installer and still uses ROCm. macOS ignores the flag (Metal, no Vulkan prebuilt).Why read from llama.cpp instead of parsing text
There is an earlier draft (#4874) that solves the same VRAM-reading problem by running
vulkaninfoand parsing its text output for memory budgets. Reading from llama.cpp's library instead is better because:vulkaninfooutput is meant for humans and changes between versions, so parsing it is fragile. We call functions and read numbers.GGML_VK_VISIBLE_DEVICESexpects.Related issues & PRs
UNSLOTH_FORCE_VULKAN; fine-tuning is not part of this change.)vulkaninfo, and it covers the whole flow rather than just detection.Testing notes
nvidia-smiwithin normal allocation noise, and the GPU order matches llama.cpp's Vulkan order (which is different from nvidia-smi's order).--simple-policy) planner, confirming Vulkan is picked first with CPU as the fallback, and that NVIDIA, ROCm, and no-GPU hosts are unaffected.UNSLOTH_FORCE_VULKANwas tested through the same--simple-policyplanner: a forced AMD host on Linux and Windows resolves to the upstream Vulkan asset (with CPU fallback), while the same host without the flag still resolves to its ROCm build.