mlxd

A fast, modular text-generation server for Apple Silicon, built in Zig on top of MLX.

mlxd provides an OpenAI-compatible API for running large language models locally with native Metal acceleration. It is a focused rewrite of the text-generation core from mlx-serve, stripped down to do one thing well.

Features

• OpenAI-compatible /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models endpoints
• Streaming (SSE) and non-streaming responses
• Chat templating via vendored jinja engine
• Tool calling with structured output
• Thinking/reasoning model support
• Speculative decoding (PLD n-gram, Gemma drafter, Qwen MTP)
• Constrained decoding (JSON schema, grammar, regex)
• Prefix caching and tokenize caching
• KV-cache quantization
• HuggingFace model download and management

Supported models

gemma3, gemma4, qwen2, qwen3, qwen3.5, llama, mistral, lfm2, nemotron_h, bert, deepseek_v4, and more.

Requirements

• macOS with Apple Silicon (M1+)
• Zig >= 0.16
• MLX >= 0.31.2 and mlx-c >= 0.6.0 (via Homebrew: brew install mlx mlx-c)

Build

zig build                          # debug build
zig build -Doptimize=.ReleaseFast  # release build

Usage

Serve

Start an OpenAI-compatible API server:

mlxd serve --model mlx-community/gemma-3-4b-it-4bit --port 8080

Then use any OpenAI-compatible client:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/gemma-3-4b-it-4bit",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": true
  }'

Run

One-shot or interactive text generation:

mlxd run --model mlx-community/gemma-3-4b-it-4bit "Explain quicksort in one paragraph"
mlxd run --model mlx-community/gemma-3-4b-it-4bit --chat

Pull

Download a model from HuggingFace:

mlxd pull mlx-community/gemma-3-4b-it-4bit

List

List locally available models:

mlxd list

Architecture

mlxd is structured as small Zig modules with compile-enforced dependency direction:

core        - shared types, OpenAI DTOs, config, logging
mlxbridge   - mlx-c FFI (sole GPU interface)
chat        - template rendering, tool-call parsing (pure, no mlx)
model       - config loading, tokenizer, model discovery
engine      - inference actor, generation, speculative decoding, caching
registry    - HuggingFace download/resolve (no mlx)
http        - server, router, SSE, API handlers
cli         - subcommand dispatch

A single engine actor thread owns all GPU state. HTTP handlers never touch mlx directly - they communicate via typed message passing and receive token streams.

License

MIT