mlx-quant-bench

Quantization benchmarks for LLMs on Apple Silicon via MLX.

Compare loaded memory, time-to-first-token, decode throughput, and output quality across precision levels (bf16 / 8-bit / 4-bit / 3-bit) on a single Mac. Works with Mistral 7B Instruct v0.3 and any model in the mlx-community org on Hugging Face.

Why this exists

Quantization is the single most-effective production optimisation for self-hosted LLMs: typically a 4× memory reduction at a marginal quality cost. But the trade-offs are model and task-specific, and the canonical tutorials don't actually show you the numbers on your hardware. This repo runs the comparison end-to-end on your Mac and produces a report you can read in five minutes.

The benchmark distinguishes three prompt categories - factual, reasoning, instruction-following - because precision degradation hits each one differently. Factual recall holds up to 4-bit easily; multi-step reasoning starts breaking earlier; instruction-following format compliance is often the first thing to go at 3-bit.

On the headline run with Mistral 7B Instruct v0.3, the surprising finding was that 4-bit outperformed 8-bit on speed without any visible quality cost: quantization on Apple Silicon turns out to be a strictly memory-and-speed win for this model, not a quality trade-off.

Example output (Mistral 7B Instruct v0.3 on M3 Pro / 36 GB)

Model	Peak memory	Load time	Avg TTFT	Avg tok/s
bf16	14.60 GB	4.0s	0.66s	8.5
8bit	7.86 GB	1.6s	0.28s	15.1
4bit	4.29 GB	0.9s	0.23s	27.0
3bit	3.40 GB	0.6s	0.23s	32.9

Three findings worth flagging:

1. Memory and speed scale together, almost linearly. Each precision step roughly halves both memory and time-per-token. That's not coincidence — it's empirical confirmation that memory bandwidth is the dominant bottleneck for token generation on M-series GPUs, not compute. Quantization on Apple Silicon is a strict win on both axes simultaneously.

2. Quality degradation hits instruction-following before factual recall. Across the prompt suite, factual prompts and reasoning chains held up to 3-bit. The first visible breaks were on bullet_summary (3-bit produced 5 bullets when asked for exactly 3) and negation (3-bit produced self-contradicting output, calling the platypus both "not a mammal" and "a mammal" in the same answer). That's the textbook narrative made concrete.

3. Single-sample variance dominates precision-induced quality differences on this prompt suite. On the age_puzzle reasoning prompt, bf16 and 8-bit both got the answer wrong (equation setup error) while 4-bit and 3-bit got it right. That's almost certainly stochastic, not precision-related, and it's a clean reminder that rigorous quality assessment of quantization needs many samples per prompt and statistical tests, not eyeball comparison on a 10-prompt suite.

See docs/REPORT_full_models.md for the full per-prompt outputs side-by-side.

The docs/REPORT_m3pro.md contains the report run on the same hardware but limited to the prebuilt Mistral7B models as provided by the mlx_community on Hugging Face (i.e. 8bit and 4bit).

Quick start

# 1. Clone and install
git clone https://github.com/leriomaggio/mlx-quant-bench.git
cd mlx-quant-bench
pip install -e .

# 2. Set up Hugging Face access (one-time, see "Hugging Face setup" below)
hf auth login

# 3. Inspect your hardware
mlx-quant-bench inspect

# 4. Sanity-check with TinyLlama (~3 GB download, ~2 minute run)
mlx-quant-bench run --preset tinyllama --reset
mlx-quant-bench report results/runs.jsonl

# 5. The headline run: Mistral 7B at two precisions (~12 GB download, ~10 min)
mlx-quant-bench run --preset mistral7b-prebuilt --reset
mlx-quant-bench report results/runs.jsonl --md REPORT.md --show-outputs

That's the full Plan A workflow. The report markdown file is what you'd commit alongside results.

Two benchmark plans

Plan A — pre-built models (default, fastest)

Uses the 4-bit and 8-bit Mistral 7B variants pre-published on mlx-community. No conversion step, no special setup beyond Hugging Face login. The downside: only two precisions, which gives you a meaningful but narrow comparison.

mlx-quant-bench run --preset mistral7b-prebuilt --reset

This is the recommended path for first-run, for portfolio demonstrations, and for anyone who wants results in under 15 minutes.

Plan B — full precision ladder (bf16, 8-bit, 4-bit, 3-bit)

Covers four precisions across the full quantization spectrum. The catch: mlx-community doesn't pre-publish bf16 or 3-bit variants for Mistral 7B — they need to be locally converted from the original Mistral release using mlx_lm.convert. The repo includes a helper script that does this in one command:

# One-time: locally convert bf16 and 3-bit (~10 min, ~28 GB disk)
bash scripts/convert_models.sh

# Run the full four-precision benchmark
mlx-quant-bench run --preset mistral7b-full --reset
mlx-quant-bench report results/runs.jsonl --md REPORT.md --show-outputs

Plan B is what you'd use if you want a complete memory ladder for a blog post or a thorough quality assessment. The conversion is a one-time cost; subsequent benchmarks reuse the cached models.

Custom model specifications

If neither preset fits, specify models directly with --model LABEL=SOURCE. Each spec pairs a precision label (free-form, used in the report) with a source (Hugging Face repo ID or local path). Repeatable:

mlx-quant-bench run \
  --model 4bit=mlx-community/Mistral-7B-Instruct-v0.3-4bit \
  --model 8bit=mlx-community/Mistral-7B-Instruct-v0.3-8bit \
  --model my-experiment=./models/my-fine-tune

This is also the way to benchmark non-Mistral models — point at any mlx-community repo or a local MLX-format directory.

Hugging Face setup

The benchmark downloads models from Hugging Face. Even though the mlx-community org is fully open, Hugging Face requires authenticated access for downloads as of late 2025.

Step 1: Create an account

If you don't have one: huggingface.co/join. Free tier; no payment details needed.

Step 2: Generate an access token

1. Go to huggingface.co/settings/tokens.
2. Click "Create new token".
3. Choose Read as the token type (minimum required, safest).
4. Give it a memorable name (e.g. mlx-quant-bench).
5. Copy the token immediately — it starts with hf_... and is shown only once. Treat it like a password.

Step 3: Log in via CLI

hf auth login

Paste the token when prompted (it won't show on screen — that's normal). The "Add token as git credential?" prompt: either answer is fine for this workflow.

The CLI is hf as of late 2025. The older huggingface-cli binary is deprecated.

Step 4: Verify

hf auth whoami

Should print your username. You're set — mlx_lm.load() will pick up the token automatically.

Alternative: environment variable

If you'd rather not run hf auth login:

export HF_TOKEN="hf_xxxxxxxxxxxxxxxxxxxxxxxx"

Add to ~/.zshrc for persistence.

Optional: pre-download models

To avoid the model download happening during the benchmark itself:

# TinyLlama (sanity check, ~2 GB)
hf download mlx-community/TinyLlama-1.1B-Chat-v1.0-mlx

# Mistral 7B variants for Plan A (~12 GB total)
hf download mlx-community/Mistral-7B-Instruct-v0.3-4bit
hf download mlx-community/Mistral-7B-Instruct-v0.3-8bit

Models cache in ~/.cache/huggingface/hub/. To redirect the cache to a larger disk:

export HF_HOME="/Volumes/BigDisk/.cache/huggingface"

Set this before both downloading and running the benchmark.

Hardware requirements

This is Apple Silicon only — MLX uses Metal and won't run on Intel Macs. Approximate memory budgets for full benchmarks:

Mac config	Plan A (4bit + 8bit)	Plan B (full ladder)
16 GB unified memory	✅ tight, close other apps	❌ won't fit bf16
24 GB unified memory	✅ comfortable	⚠️ tight on bf16
36+ GB unified memory	✅ comfortable	✅ comfortable

Run mlx-quant-bench inspect to see your specific budget and a model-fit table.

Repository structure

mlx-quant-bench/
├── src/mlx_quant_bench/
│   ├── __init__.py        # Public API: ModelSpec, parse_spec, expand_preset
│   ├── cli.py             # `mlx-quant-bench` entry point
│   ├── spec.py            # Model spec parsing and presets
│   ├── prompts.py         # Benchmark prompt suite (factual/reasoning/instruction)
│   ├── benchmark.py       # Load model, run prompts, record metrics
│   ├── compare.py         # Read JSONL, render reports
│   └── inspect.py         # Hardware introspection
├── scripts/
│   └── convert_models.sh  # Plan B helper: locally convert bf16 + 3-bit
├── tests/
│   ├── test_spec.py       # Spec parsing and preset definitions
│   ├── test_prompts.py    # Prompt suite invariants
│   └── test_compare.py    # Report rendering
├── pyproject.toml
├── LICENSE
└── README.md (this file)

What gets measured

For each prompt:

• TTFT (time to first token) — bounds chat-app responsiveness.
• Decode throughput (tok/s) — bounds long-form generation cost.
• Output tokens — for normalising throughput across prompts of different lengths.
• Full output text — for eyeball quality assessment.

Per-model:

• Load time — time to read weights from disk and initialise.
• Peak GPU memory — bounds what fits during inference.

All metrics land in a JSONL file, one record per prompt. The report subcommand turns this into a markdown summary with optional side-by-side outputs for visual quality comparison.

Implementation notes

A few choices worth flagging:

• MLX is the only backend. No PyTorch, no bitsandbytes. This is a Mac-native benchmark for Mac-native deployment patterns.
• TTFT is approximated by running two generate() calls — one with max_tokens=1 for the first-token timing, then a full generation for decode throughput. This isn't perfect (it double-counts the prefill), but it's robust across mlx-lm versions.
• MLX peak memory is reset between models via mx.metal.reset_peak_memory() to prevent cross-contamination.
• The model spec system decouples "where to load from" from "what to call it". This matters because mlx-community doesn't follow a uniform -{precision} suffix pattern across models — Mistral has 4-bit and 8-bit variants pre-published, TinyLlama has only a single MLX-format repo, bf16 / 3-bit Mistral need to be locally converted. The CLI handles all three cases through one interface.

Running tests

pip install -e ".[dev]"
pytest

The tests cover spec parsing, preset definitions, prompt invariants, and report rendering. They don't require MLX (those modules are imported lazily), so the test suite runs on any Python ≥3.10.

License

MIT — see LICENSE.

Author

Valerio Maggio, Senior Developer Advocate. github.com/leriomaggio · linkedin.com/in/valeriomaggio