RuntimeRouter

Optimize the entire AI session, not just the next model call.

RuntimeRouter is a Python-first LLM routing library for the AI ecosystem. It sits above agent frameworks — LangGraph, PydanticAI, CrewAI, AutoGen — as a Runtime Router, not a replacement for them.

You write agents with your favorite framework. RuntimeRouter decides which model to call, when, and why — across the full session lifecycle.

---

Vision

Traditional routers optimize a single inference call: given a prompt, pick the cheapest or fastest model.

RuntimeRouter's long-term goal is different: optimize the entire AI session — model selection, cost, latency, context, caching, and privacy — as a unified runtime decision layer.

┌─────────────────────────────────────────────────────────┐
│  Your Agent Framework (LangGraph / PydanticAI / …)      │
├─────────────────────────────────────────────────────────┤
│  RuntimeRouter  ← session-aware routing (this library)  │
├─────────────────────────────────────────────────────────┤
│  LiteLLM / Provider APIs (OpenAI, Anthropic, …)         │
└─────────────────────────────────────────────────────────┘

---

Why RuntimeRouter?

Problem	RuntimeRouter approach
Hard-coding model="gpt-4o" everywhere	AutoLLM() picks the right model per request
Simple tasks burning frontier-model budget	Complexity routing sends easy tasks to cheap models
No visibility into routing decisions	Every call returns a RouteDecision with reason
Framework lock-in	Framework-agnostic; works with any LiteLLM-compatible stack
Proxy-only routers	Designed for session-level optimization (roadmap)

---

vs OpenRouter

	OpenRouter	RuntimeRouter
What it is	Unified API proxy to 100+ models	Python routing library embedded in your app
Scope	Single HTTP request → model	Entire AI session (roadmap)
Integration	Replace your API base URL	Drop-in AutoLLM() or Router in Python code
Policies	Provider-side routing rules	Pluggable Python Policy classes you own
Framework	Language-agnostic HTTP	Python-first, native LangGraph/PydanticAI hooks (v0.2+)

OpenRouter is excellent as a model gateway. RuntimeRouter is a runtime decision engine you control in-process.

---

vs Not Diamond

	Not Diamond	RuntimeRouter
What it is	Managed routing SaaS	Open-source Python library
Deployment	Cloud API	In-process, self-hosted
Customization	Platform-configured	Full plugin Policy architecture
Session scope	Per-call model selection	Session-aware optimization (roadmap)
Cost	SaaS pricing	Free & open (MIT)

Not Diamond provides intelligent routing as a service. RuntimeRouter gives you the same concept as extensible, auditable Python code.

---

Quick Start

Install

pip install runtimerouter

Basic usage

from runtimerouter import AutoLLM

llm = AutoLLM()

# RuntimeRouter automatically selects Claude, Gemini, DeepSeek, OpenAI, etc.
response = llm.invoke("帮我分析整个代码仓库")

print(response.choices[0].message.content)

Inspect routing decisions

decision = llm.route_only("Summarize this paragraph in one sentence.")
print(decision.selected_model)  # e.g. "gpt-4o-mini"
print(decision.reason)          # why this model was chosen
print(decision.complexity)      # e.g. ComplexityLevel.SIMPLE

Configure policies

from runtimerouter import AutoLLM, RouterConfig

config = RouterConfig(
    enable_complexity_routing=True,
    enable_cost_routing=True,
    fallback_model="gpt-4o-mini",
    policy_order=["complexity", "cost"],
)

llm = AutoLLM(config=config)

Use Router directly (without invoking)

from runtimerouter import Router, RouteContext

router = Router()
decision = router.route(RouteContext(prompt="Explain quantum entanglement"))
print(decision.selected_model)

---

Architecture

runtimerouter/
├── autollm.py          # User-facing entry point
├── router.py           # Routing pipeline orchestrator
├── classifier.py       # Task complexity & token estimation
├── types.py            # Shared data contracts (Pydantic models)
├── config.py           # Config loading utilities
├── policies/           # Pluggable routing policies
│   ├── base.py         # RoutingPolicy ABC
│   ├── complexity.py   # Complexity-based routing
│   └── cost.py         # Cost-based routing
├── providers/          # Model catalog & provider metadata
│   ├── base.py         # ModelProvider ABC
│   └── registry.py     # ModelRegistry (default model catalog)
└── integrations/       # External execution layers
    └── litellm.py      # LiteLLM completion wrapper

Routing pipeline (v0.1):

User prompt
    │
    ▼
TaskClassifier.enrich()     ← complexity, token estimate
    │
    ▼
Policy chain (ordered)      ← complexity → cost
    │
    ▼
RouteDecision               ← selected model + reason
    │
    ▼
LiteLLMIntegration          ← execute completion

See docs/architecture.md for module boundaries and extension points.

---

Roadmap

Version	Focus
v0.1 (current)	Auto Model Selection, Complexity Routing, Cost Routing, LiteLLM Integration
v0.2	LangGraph Integration, PydanticAI Integration
v0.3	Cache-aware Routing
v0.4	Context-aware Routing
v0.5	Session-aware Optimization

Full details: ROADMAP.md

---

Contributing

We welcome contributions! See CONTRIBUTING.md.

git clone https://github.com/chenyu-dev25/RuntimeRouter.git
cd RuntimeRouter
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest

---

License

MIT — see LICENSE.

---

RuntimeRouter
Optimize the entire AI session, not just the next model call.