metal: fall back to copied weight buffers on macOS 14 paravirtual devices#37
Open
danielhanchen wants to merge 479 commits into
Open
metal: fall back to copied weight buffers on macOS 14 paravirtual devices#37danielhanchen wants to merge 479 commits into
danielhanchen wants to merge 479 commits into
Conversation
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
…5045) Tests are generally prefixed with -test, so rename export-graph-ops accordingly. rpc-server is probably too generic a name for /usr/bin. Because it should work with any ggml application, it is renamed to ggml-rpc-server.
…#25057) * [CUDA] Added a cudaMemcpy2DAsync fast path to ggml_cuda_cpy Add a CUDA ggml_cpy fast path for same-type, same-shape strided copies that are just 2D pitched block copies. When tensors are not fully contiguous but each row is contiguous, it now uses cudaMemcpy2DAsync instead of the slow element-wise scalar copy kernel. This fixes the GDN recurrent snapshot update with -np 4, where rollback slots are separated by cache stride gaps. * Add new tests that execute the new optimized strided copy path * Return unsupported for strided copy in OpenVINO, as new tests are failing
* opencl: rework FA kernel for f16 and f32
* opencl: flash-attention prefill prepass kernels
- flash_attn_kv_pad_f16 pads the tail KV tile to a BLOCK_N multiple
- flash_attn_mask_pad_f16 pads the matching mask tile
- flash_attn_blk_f16 classifies each KV tile per query block as
fully masked / mixed / fully unmasked, so
the main kernel can skip fully-masked tiles
and the mask lookup for fully-unmasked ones
* opencl: FA kernels for q4_0 and q8_0
* opencl: `set_rows` for f32 to q8_0/q4_0
* opencl: dequant kernels for q4_0 and q8_0
* opencl: add FA tile tuning table with override
* opencl: wire host side for FA
* opencl: q4_0 MoE tensors are also SOA'ed
* opencl: cosmetic fix
* opencl: refactor, also clarify some code paths in comments
* opencl: fix inifity for `-cl-finite-math-only`
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* server : reduce logs * cont : common * cont : spec * cont : CMN_ -> COM_
Expose the existing --offline flag to `llama download` so a script can run it to check whether a model is already cached and ready to be served without touching the network. Also fix a latent use-after-free in the URL-task on_done callback: first_path is block-scoped and was captured by reference, but invoked after the block ends. Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* spec: add DFlash v2 support * dflash: support sliding window attention per layer_types * docs: add dflash section --------- Co-authored-by: Kashif Rasul <kashif.rasul@gmail.com>
* jinja: add --dump-prog for debugging * Update common/jinja/runtime.cpp Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> --------- Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* Add minicpm5 tool call parser * Refactor MiniCPM5 PEG parser per review feedback * Fix jinja min/max API to match Jinja2 * modify by review * MiniCPM5: use autoparser for XML tool calls and fix grammar preserved-token triggers * MiniCPM5: fix streaming tool-arg placeholder and remove alt XML markers * skip min/max attribute tests in -py mode * test-jinja: use real expected output for min/max attribute tests * MiniCPM5: revert shared mapper and history fallbacks per review Drop streaming tool-arg placeholder workarounds from the generic PEG mapper and restore strict tool-call argument JSON parsing so MiniCPM5 support stays limited to autoparser/diff-analyzer changes. * chat : refactor minicpm5 back to dedicated parser * cont : simplify grammar * cont : refactor * cont : fixes * cont : rename template to openbmb-MiniCPM5-1B.jinja * cont : add message delimiters * cont : fix tests --------- Co-authored-by: zhangtao <zhangtao2@modelbest.cn> Co-authored-by: 张涛 <>
* dflash: refactor draft model conversion * apply fix for eagle3 convert
…sisted by claude(in debugging and tests) (ggml-org#24727)" (ggml-org#25098)
* jinja, chat: add --reasoning-preserve flag * correct help message
* convert: add dsv4 conversion * add basic setup * add llm_graph_input_dsv4 * add save-load state * add sinkhorn eps - correction by @fairydreaming * add rope fix * cleanup dead code * fix bugs * support pro model: added by @fairydreaming * remove redundant V cache * Chat template * remove debugging leftovers * Add mechanism for inlining templates based on architecture * s/deepseek-v4-flash/deepseek4/g * s/deepseek-v4-flash/deepseek4/g continued * enable graph reuse * enable FA * fix test llama archs * rename * compatibility with antirez ds4 GGUFs * simplified set_gguf_parameters() by calling super class method, replaced moe.score_func with expert_gating_func. * reserve worst-case kv-cache * revert max split inputs * address review comments * add padding to enable FA * pad only the final value of plan.n_kv to 256 * remove built-in cpp chat template * cont: remove cpp built-in template * rm outdated test * replace ggml_view_3d() with ggml_reshape_3d() Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * only support n_seq=1 for now * remove unused var * cont: remove unused var * use scale bias * use correct ptr for can_reuse * remove gen-chat-inline-templates.py * simplify graph reuse * cont: cleanup * remove unused inputs * enable partial checkpointing * add correct shape for kq_mask + set llama_model_n_swa to 0 for dsv4 * precompute source_idx + add comment about dummy write * support multi-seq * remove restored_trim_pos * use split_equal when possible * fix indent * address review comments * use LLM_KV * fix ci --------- Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com> Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
…tic (ggml-org#25005) * vulkan: extract flops calculation into function * use flops instead of matmul src0 tensor size for submission threshold * use unsigned ints
…g#25131) Signed-off-by: Adrien Gallouët <angt@huggingface.co>
…org#24588) * HIP: keep MMQ for gfx900 MoE and Q8_0, use hipBLAS for dense K-quants Assisted-by: GitHub Copilot CLI * HIP: tighten conditional block to be explicitly for gfx900 * HIP: Further simplified gfx900 conditional block * removed unnecessary comment
* vulkan: roll bk loop in matmul for asahi linux * vulkan: fix inline comment * vulkan: revert BK-loop unroll change * vulkan: edit spirv directly for asahi roll bk loop * vulkan: remove trailing whitespace at the end of comments
* CUDA: fix Gemma E4B MTP FlashAttention * remove unused template declaration
…org#25140) * common,server: handle bracketed IPv6 literals in URL authority Parse the [host]:port form (RFC 3986) and bracket IPv6 hosts when formatting a URL authority: listening log, proxy Host header, proxy log, client rebuild. The per-request remote_addr stays bare. * common: restore unsupported scheme throw in url parser Address @ngxson review: keep the explicit reject in port resolution so the block stays self-contained. Non-http(s) schemes still throw (also gated at the top of common_http_parse_url).
* Fix input assignment in layer processing loop Fix DFLASH for qwen-coder-next * add line break Added tensor for attention normalization in Qwen3 model.
* support cuda virtual devices * disable NCCL path when virtual devices are used * label virtual devices in description; add GPUx2 server CI jobs * code refactor
…l-org#25733) * spec: fix dflash target tokenizer mismatch during conversion * fix ci ty check
Microsoft BitNet Hugging Face configs use BitNetForCausalLM while the converter only registered BitnetForCausalLM, causing conversion to fail with "Model BitNetForCausalLM is not supported". Register both spellings in TEXT_MODEL_MAP and the Bitnet model class. Fixes ggml-org#25629
) The current integration treats SME as a single capability (CPU_FEATURE_SME) with no distinction between SME(v1) and SME2. The kernels dispatched under CPU_FEATURE_SME use SME2-specific instructions, making dispatch incorrect on SME(v1)-only hardware. We introduce build-time and runtime distinction between SME and SME2, and wire SME(v1) and SME2 kernels based on actual hardware support.
…ing) and more MUL_MAT updates (ggml-org#25762) * hex-mm: fix artificial limit in the solver that restricted number of act-prep threads * hex-mm: fix warning * hex-prof: do not apply --top to the timeline report * hmx-mm: add suport for tiled act-processing to better distribute hvx work * hex-l2: add tracing for l2flush events * workqueue: redo the legacy workpool api to match hmx-queue and dma-queue * hmx-mm: fix f32 activation buffer alignmnet for nhvx=5,6,7 * hex-work: minor cleanup for work-queue apis * hex-work: further cleanup of the work-queue api * hex-l2: optimize l2flushes at the opbatch level * hex-work: remove unused mask * hex-work: no need to drop hvx ctx in the work-queue * hex-work: add explicit wakeup/suspend and make threads spin * hex-bufs: mark any non-weight tensor as compute * hex-dma: dma-queue support for alias queues and cached dma * hex-l2: track tensor aliases and delay or skip flushes as much as possible * hex-l2: simplify tensor alias handling * hex-l2: handle overlapping views as a circular list of aliases * hex-tens: add flags helper * hex-l2: add helper for marking tensors clearn/dirty * hex-l2: mark binary and rope outputs as l2-clean and keep the rest as is for now * hex-l2: proper support for handling all tensor overlap scenarios * hex-trace: instrument matmul init code and cleanup trace checks * hex-thread: introduce dedicated main thread with explicit stack and priority * hex-l2: track dirty state as bitmap and introduce threaded flush * hex-trace: remove redundant checks for ctx != null * hex-l2: allocate entire context as one buffer and l2fetch it after big flushes * hex-l2: disable tensor clearing in binary and rope for now seems to cause issues with fusion * hmx-mm: update act proc to use fastdivs and fix DMA overflow * hmx-mm: make MUL_MAT_ID kernels robust to multi-chunk cases (start_row>0) * hex-queue: remove obsolete queue interfaces and flush hmx-queue at the end of the op-batch * hex-queue: dont use early wakeup for small op-batches * hex-tensors: properly cap max_tensors in op-batches and dirty_map * hex-l2: make sure threaded l2flush does proper rounding * hex-l2: factor out htp_tensor_flush for reuse (if needed) * hex-l2: optimize tensor flushes by coalescing flush-all * hex-l2: optimize multi-threaded flush * hex-drv: futureproof version checks * hexagon: fix errors and warnings on windows * hex-main: update main thread to only use dspqueue_read, dspqueue_peek is not available on some platforms * hex-main: add fallback mode for dspqueue with callbacks * hex-main: introduce fallback mode for using dspqueue callbacks for full op processing * hex-main: remove early wakeup, not helping and seems to cause some errors with certain batch sizes * hex-l2: make sure to use invalidate version of flushall * hex-l2: dont try to trace early l2flush at the start of op-batch * hex-main: remove offset_ctx that must be zero anyway * hex-hmx: fix hmx_queue_depth to use idx_write - idx_read * hex-hmx: use atomic_load for idx_read/write * hex-main: add static assert to make sure n_threads are aligned
* dsv4 hc-ops * add missing files; * add cparams * update rpc version * address review comments * address review comments
…for Adreno A7x GPUs) (ggml-org#25780) * opencl: load quant as uint in mv_q4_k_f32_flat helps older compilers (e.g., E031.41, boosts 2x), no impact on newer compilers (e.g., E031.45 or newer) * opencl: load quant as uint in mv_q5_K_f32_flat helps older compilers (e.g., E031.41) * opencl: format --------- Co-authored-by: Li He <lih@qti.qualcomm.com>
…25690) * sycl: fix incorrect row calculation when K_QUANTS_PER_ITERATION=1 Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * sycl: use K_QUANTS_PER_ITERATION for non-reordered Q5_K kernel This is the only Q5_K kernel that was not using KQPI. Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * sycl: add missing second half processing to reordered q5_k Error found while running GGML_SYCL_PRIORITIZE_DMMV=1 \ build/bin/test-backend-ops test -o MUL_MAT Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * sycl: fix potential off-by-one error Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * sycl: fix missing row > nrows check Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> --------- Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* vulkan: Support Q2_0 The backend perf tests for mat-vec-mul weren't very good at first (worse than q2_k), doubling the rows per workgroup made a big difference. * reorder * resolve merge conflict, adjust err threshold for f16->q2_0 set_rows
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
* common: auto-download dflash- and eagle3- HF sidecars Mirror the existing mtp- sidecar logic to support auto-discovery and download of DFlash (dflash-) and Eagle3 (eagle3-) speculative decoding sidecars from Hugging Face repos. Changes: - Add --dflash and --eagle3 CLI flags to trigger sidecar download - Add find_best_dflash() and find_best_eagle3() using find_best_sibling - Exclude dflash- and eagle3- filenames from primary model selection - Filter dflash- and eagle3- from cached model listings - Wire download tasks that set speculative.draft.mparams as fallback Assisted-by: pi:llama.cpp/Qwen3.6-27B * docs : regen
…nel tensors (ggml-org#25822) Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
…it (vectorized LD/ST perf opt) for Adreno GPUs (ggml-org#25810) * opencl: read MoE dp4a activation tile as 128-bit local loads * opencl: vectorize MoE dp4a activation staging as 128-bit loads
…ation (ggml-org#25787) DeepSeek-V4's ffn_gate_tid2eid tensor is an i32 token-id -> expert-id index table, not weights. It was never added to the name-based exclusion list alongside ffn_gate_inp.weight, so llama-quantize tries to quantize it and fails since i32 cannot convert to a float type. Fixes ggml-org#25754 Signed-off-by: Yash Raj Pandey <yashpn62@gmail.com>
* dflash: rotate injected K/V cache when using K/V quantization * Update src/models/dflash.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * clearer format * remove trailing whitespace --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
…r Adreno Q8_0 MUL_MAT for llama-server multi-stream (ggml-org#25910) * opencl: handle broadcast for adreno gemm/gemv_noshuffle * opencl: honor view_offs for adreno noshuffle gemm/gemv * opencl: general GEMM/GEMV support broadcast * opencl: remove unnecessary tests * opencl: remove unnecessary comments --------- Co-authored-by: Li He <lih@qti.qualcomm.com>
…devices The Apple Paravirtual device exposed inside macOS 14 virtual machines (GitHub macos-14 hosted runners among others) returns stale data when a Metal buffer is created with newBufferWithBytesNoCopy over a file-backed mmap region. Model weights read as garbage from the GPU and inference produces degenerate output. The same binary is correct on macOS 15+ VMs, on bare metal, and with --no-mmap or -ngl 0. Detect that combination at device init (Paravirtual device name and macOS < 15) and stop advertising the buffer_from_host_ptr capability so llama.cpp falls back to copied weight buffers. GGML_METAL_MMAP_BUFFERS_DISABLE forces the fallback for debugging.
danielhanchen
added a commit
that referenced
this pull request
Jul 23, 2026
* Prebuilt: carry macOS 14 paravirtual mmap fix; drop stale GGML_METAL_USE_BF16 define Pin #37 in pr-set.json: on macOS 14 VMs the Apple Paravirtual GPU returns stale data for mmap-backed weight buffers, so the patch makes the Metal backend stop advertising buffer_from_host_ptr there and llama.cpp falls back to copied weights (same path as --no-mmap). GGML_METAL_USE_BF16 no longer exists in current llama.cpp (bf16 is runtime-probed since the ggml-metal rewrite), so the define was a no-op; remove it from the macOS matrix entry. * Keep the pin-set unchanged; this PR only drops the stale define GGML_METAL_USE_BF16 no longer exists as a build option in the current source tree (bf16 is runtime-probed), so passing it was a no-op. The macOS 14 paravirtual mmap pin moves to a separate decision.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Carried patch for the prebuilt nightlies, to be pinned in scripts/unsloth/pr-set.json. The branch is based on upstream tag b10069 so it merges cleanly onto current base tags; the diff against master shows unrelated upstream commits, the actual change is the single commit c417c7f touching only ggml/src/ggml-metal.
Problem: the published macOS arm64 bundles produce degenerate all empty output for unsloth/gemma-4-E2B-it-GGUF UD-Q4_K_XL via Metal on macOS 14 VMs (GitHub macos-14 runners). The server loads, /health is 200, completions are empty.
Root cause: the Apple Paravirtual device of macOS 14 VMs returns stale data when Metal reads weights through a buffer created with newBufferWithBytesNoCopy over a file-backed mmap region. Diagnostics on real runners:
Fix: at Metal device init, when the device name contains Paravirtual and the OS is macOS < 15, stop advertising the buffer_from_host_ptr capability. llama.cpp then falls back to copied weight buffers (same path as --no-mmap), which are correct there. Real hardware and macOS 15+ VMs are unaffected. GGML_METAL_MMAP_BUFFERS_DISABLE forces the fallback for debugging.
Repro runs: https://github.com/danielhanchen/unsloth-staging-2/actions/runs/29924253302 and https://github.com/danielhanchen/unsloth-staging-2/actions/runs/29925644369