Skip to content

UPSTREAM PR #18042: model: support GLM4V vision encoder#570

Open
loci-dev wants to merge 27 commits into
mainfrom
upstream-PR18042-branch_ngxson-xsn/glm4v
Open

UPSTREAM PR #18042: model: support GLM4V vision encoder#570
loci-dev wants to merge 27 commits into
mainfrom
upstream-PR18042-branch_ngxson-xsn/glm4v

Conversation

@loci-dev

Copy link
Copy Markdown

Mirrored from ggml-org/llama.cpp#18042

On first look, it seems to be an easy model to support as the HF implementation is pretty much the same as Qwen2.5VL

However, there are some very subtle differences that even some LLMs will miss (I tried both Grok and Gemini 3 and they both missed the 2 first points):

  1. For text model, M-RoPE ordering is non-Neox. And because ggml's M-RoPE uses the Neox ordering by default, I had to add a new flag to disable this --> need to be supported on backends, I'm only including CPU and Metal in this PR. This is by far the most complicated change to support this model
  2. Learned position embedding interpolation uses bicubic instead of bilinear
  3. Added a norm layer right after patch bias
  4. Use RMS norm

The embedding output was tested against HF transformers and confirmed to be matched

Testing

I'm using the ./tools/mtmd/test-1.jpeg already included in this repo:

llama-mtmd-cli -m ..... -mm ..... --image ./tools/mtmd/test-1.jpeg -p "extract all texts from this image" --temp 0 -n 1024

Output:

"All the News That's Fit to Print"

Then the newspaper title: "The New York Times"

"LATE CITY EDITION"

"VOL. CXLVIII, No. 40,711"

"NEW YORK, MONDAY, JULY 21, 1969"

"10 CENTS"

Then the main headline: "MEN WALK ON MOON"

Next: "ASTRONAUTS LAND ON PLAIN; COLLECT ROCKS, PLANT FLAG"

Then a section: "Voice From Moon: 'Eagle Has Landed'"

Then the article by John Noble Wilford: "A Powdery Surface Is Closely Explored"

Now, let's transcribe each part carefully, including smaller text.

First, the top left box:

"All the News  
That's Fit to Print"

Then the newspaper header:

"The New York Times"

"LATE CITY EDITION"

"VOL. CXLVIII, No. 40,711"

"NEW YORK, MONDAY, JULY 21, 1969"

"10 CENTS"

@loci-dev
loci-dev force-pushed the main branch 4 times, most recently from d582acc to 4b559d8 Compare December 15, 2025 13:21
@loci-dev
loci-dev force-pushed the main branch 4 times, most recently from 79099fb to 4664cb4 Compare December 15, 2025 19:09
@loci-dev
loci-dev force-pushed the main branch 30 times, most recently from a05d3f0 to c9b2647 Compare December 23, 2025 02:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants