KTransformers Roadmap (2026 Q2)

[ErvinXie]

KTransformers Roadmap - 2026 Q2

Focus

• Fine-tuning: Make LoRA SFT run on the exact same hardware as inference — if you can run it, you can fine-tune it.
• Consumer Hardware: Deliver the best MoE inference performance on consumer-grade x86 + NVIDIA GPU.
• Windows: Native Windows support with full heterogeneous performance advantage.
• Performance: VNNI instruction set optimization and AI SSD exploration.

Fine-tuning Service @JimmyPeilinLi

Goal: Lower the barrier for LoRA SFT to match inference — same hardware, same setup, zero extra cost.

• Release fine-tuning service to main branch (basic version first, iterate later).
• Unify inference and SFT paths so both run on one set of hardware with shared model support, weight handling, and config.
• Support transformers v4 and v5, minimum 12G VRAM for 67B model. ✅

Consumer Hardware Support

Goal: Best-in-class MoE inference performance on consumer hardware.

• Windows native C++ inference support — investigate JIT compiler Windows compatibility.
• Evaluate S1 inference backend migration to Windows as alternative path.
• Complete VNNI instruction set optimization.
• Investigate AI SSD performance bottleneck (slow mmap reads from disk).
• Promote AVX2 to first-class supported tier.
• Improve support quality and stability for RTX 30/40/50 series.

Performance

• Optimize decode and prefill paths. @ouqingliang
• Optimize expert offloading and scheduling strategies. @ovowei
• Benchmark on representative AVX2 + RTX 30/40/50 configurations. @yyj6666667
• Focus on heterogeneous performance overhead in Windows path.
• Support nvfp4 and mxfp4 quantization formats (carried over from Q1).

Coverage

• Fast support for important frontier MoE models.
• Improve support quality and completeness for major model families.

Contribution / Maintenance

• Maintain CI/CD pipelines.
• Clearer documentation
• Establish recommended configurations for typical consumer hardware setups.
• Continue NUMA-aware optimization and CPU-GPU coordination improvements.

---

Any contribution is welcomed, please email yzwliam@126.com / ervinxie@qq.com if you want to join development WeChat group.

[zklc]

nvidia发布了不少nvfp4的量化模型。有针对这块做优化吗？尤其是cpu侧支持nvfp4的高效计算。

[wanjunling168]

Windows native C++ inference support (JIT compiler) <- triton-windows probably enough?

[rnett]

nvidia发布了不少nvfp4的量化模型。有针对这块做优化吗？尤其是cpu侧支持nvfp4的高效计算。

Also interested (albeit mostly in the GPU side, since I have a blackwell GPU). It was in the Q1 roadmap, but wasn't checked off and the only related branch I see had its last commit 3 months ago.

[aliez-ren]

will be able to run rawint4 on avx2?

[ErvinXie]

@zklc @rnett Thanks for bringing this up. You're right that nvfp4 was on the Q1 roadmap but didn't get completed. We're planning to prioritize it in Q2 — along with mxfp4 support. Will update the roadmap accordingly. Stay tuned.

---

This reply was co-authored by an AI agent and a human.

[ErvinXie]

@aliez-ren If you're referring to QAT INT4 formats (like Kimi's raw INT4 weights) — we don't support those yet. The main challenge is that AVX2-only machines typically don't have enough memory bandwidth and compute throughput to run models at that scale efficiently in raw INT4.

That said, AWQ and GPTQ INT4 are fully supported today, and we recently merged VNNI-256 support for GPTQ INT4 (#1926) which improves performance on consumer Intel CPUs. If you have a specific model in mind, let us know and we can advise on the best supported path.

---

This reply was co-authored by an AI agent and a human.

[yyj6666667]

@wanjunling168 Thanks for the pointer

Triton alone wouldn't be enough though — it only covers @triton.jit kernels and torch.compile. The harder half is the CUDA
C++ / CUTLASS JIT that flashinfer drives through nvcc, which triton-windows doesn't touch.

We're currently focused on the minimum JIT verification on Windows (native MSVC + nvcc), and haven't pulled triton-windows
into the picture yet.

Thanks for pointing it out.

[aliez-ren]

@ErvinXie Thanks for your reply! My machine has 384G GDDR7 VRAM + 256G DDR4 RAM. I'm going to write a patch for running RAWINT4 on AVX2

[wanjunling168]

@wanjunling168 Thanks for the pointer

Triton alone wouldn't be enough though — it only covers @triton.jit kernels and torch.compile. The harder half is the CUDA C++ / CUTLASS JIT that flashinfer drives through nvcc, which triton-windows doesn't touch.

We're currently focused on the minimum JIT verification on Windows (native MSVC + nvcc), and haven't pulled triton-windows into the picture yet.

Thanks for pointing it out.

https://github.com/vllm-project/vllm/compare/main...SystemPanic:vllm-windows:vllm-for-windows
https://github.com/flashinfer-ai/flashinfer/compare/main...SystemPanic:flashinfer-windows:main

Seems we have some limited Windows ports for flashinfer and marlin kernels I found in this repo already (though they're meant for vllm), those marlin kernels in this repo can be good reference for you @yyj6666667 ?

[yyj6666667]

@wanjunling168 Definitely — those are good references, especially flashinfer-windows. We noticed it early on. Thanks for pointing it out.

[aliez-ren]

@aliez-ren If you're referring to QAT INT4 formats (like Kimi's raw INT4 weights) — we don't support those yet. The main challenge is that AVX2-only machines typically don't have enough memory bandwidth and compute throughput to run models at that scale efficiently in raw INT4.

That said, AWQ and GPTQ INT4 are fully supported today, and we recently merged VNNI-256 support for GPTQ INT4 (#1926) which improves performance on consumer Intel CPUs. If you have a specific model in mind, let us know and we can advise on the best supported path.

This reply was co-authored by an AI agent and a human.

created a pr to support rawint4 on avx2: #1942

[willbed]

我目前有个问题，当使用Hybird Prefill的时候，即kt-gpu-prefill-token-threshold 设为0， 此时cpu还是会加载全部权重至内存，DDR5的内存实在太贵了，目前只能用SSD swap解决这个问题。 这里是否能够只加载冷门专家至内存，我理解热门专家这个时候已经装载在显卡上，在内存里是完全没必要了。