Model request: MiMo V2.5 Series

[rodrigomatta]

The MiMo-V2.5 series introduces two open-source Mixture-of-Experts (MoE) language models MiMo-V2.5-Pro with 1.02T parameters (42B activated) and MiMo-V2.5 with 310B parameters (15B activated) both featuring a context window of one million tokens. It would be great to run MiMo V2.5 with llama.cpp!

[Lissanro]

Yes, it would be great! In case testers are needed, I can help with testing including with the larger 1T model, once I finish downloading it. I am assuming some patching will be needed to quantize and run it, and if necessary patches are provided, I can build from source and test it.

[jukofyork]

Is it not the same architecture as this:

#18328

---

https://huggingface.co/XiaomiMiMo/MiMo-V2.5

Built upon the MiMo-V2-Flash backbone and extended with dedicated vision and audio encoders

https://huggingface.co/XiaomiMiMo/MiMo-V2-Flash/blob/main/config.json

https://huggingface.co/XiaomiMiMo/MiMo-V2.5/blob/main/config.json

It's might be possible to remove the audio_config and vision_config sections and then hack convert_from_hf.py to skip those tensors (a bit like what @AesSedai did with kimi-k2.5 when it was first released).

[AesSedai]

I've tried to convert this as a MimoV2Model and it mechanically worked, dropping the audio and vision config sections is easy though yes:

        if name.startswith(("audio_encoder.", "visual.", "speech_embeddings.")):
            return

Poking at this still, the resulting GGUF had a perplexity in the hundreds of thousands so I think there's something else going on.

[jukofyork]

I've tried to convert this as a MimoV2Model and it mechanic> Poking at this still, the resulting GGUF had a perplexity in the hundreds of thousands so I think there's something else going on.

Oh, if you get anywhere and need somebody to test it then I should be able to run it - I just got 1Gbit fibre installed too, so not as painful to download.

[AesSedai]

Good news, I've got some reasonable PPL numbers out of the non-Pro version of the model for text usage.

Testing the convert on both versions of the model with the fixes now and expect to have a PR up in a few hours.

[AesSedai]

PR: #22493

Working on quants of V2.5 to publish to HF, will do V2.5 Pro later once those are done.

[AesSedai]

Preview quants: https://huggingface.co/AesSedai/MiMo-V2.5-GGUF

[AesSedai]

I got vision working and uploaded the mmproj files to HF. It's on this branch, based off of the fattn branch: https://github.com/AesSedai/llama.cpp/tree/mimo-v2.5-vision

(Once the basic support branch gets merged, I'll put up the PR for the vision and fattn and work on getting those in).

[AesSedai]

Closed by #22493

There will be a follow-up PR to add the missing FA kernels.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.