Switchyard

Switchyard is a Python proxy for LLM traffic. It routes requests across providers, translates between the OpenAI and Anthropic APIs, collects usage statistics, and lets you build typed, profile-backed routing flows with little boilerplate.

Why Switchyard? Point a coding agent such as Claude Code or Codex at an open-source model. Switchyard translates between the OpenAI Chat, Anthropic Messages, and OpenAI Responses formats, so the agent keeps speaking its native API while the request is served by vLLM, NVIDIA NIM, Ollama, or any OpenAI-compatible endpoint. The same proxy can spread traffic across several models for A/B benchmarking, signal-driven stage-router escalation, or a router you write yourself.

Launcher routing is explicit. By default, launchers use the built-in LLM-classifier router, which you tune with --weak-model, --classifier-model, --profile, and --classifier-min-confidence. Use --model X for single-model passthrough or --routing-profiles FILE for a YAML route bundle.

Features

• Protocol Translation: convert between OpenAI Chat, Anthropic Messages, and OpenAI Responses formats
• Multi-Backend Routing: random routing, LLM-as-classifier routing, signal-driven stage-router, or custom routers
• Strong Types: typed request/response containers for OpenAI, Anthropic, and Responses APIs
• Profile-Owned Routing: typed profiles own routing, backend calls, stats, and translation wiring
• One-Command Launchers: switchyard launch claude, switchyard launch codex, and switchyard launch openclaw spin up a local proxy and drop you into the target CLI
• Request Statistics: collect per-request latency, token, and cost data

Quick Start

Install from PyPI

pip install "nemo-switchyard[cli,server]"

Install from source for local use (requires uv)

git clone git@github.com:NVIDIA-NeMo/Switchyard.git
cd Switchyard
uv tool install --editable '.[server,cli]'

Install from source for contributors (requires uv)

git clone git@github.com:NVIDIA-NeMo/Switchyard.git
cd Switchyard
uv sync
uv run switchyard ...

1. Launch Claude Code, Codex, or OpenClaw through Switchyard

Create an OpenRouter account at openrouter.ai and generate an API key from the OpenRouter keys page, then export it:

export OPENROUTER_API_KEY="your-openrouter-key"  # pragma: allowlist secret
export OPENROUTER_BASE_URL="https://openrouter.ai/api/v1"
switchyard launch claude --model openai/gpt-4o-mini --api-key "$OPENROUTER_API_KEY" --base-url "$OPENROUTER_BASE_URL"
switchyard launch codex --model openai/gpt-4o-mini --api-key "$OPENROUTER_API_KEY" --base-url "$OPENROUTER_BASE_URL"
switchyard launch openclaw --model openai/gpt-4o-mini --api-key "$OPENROUTER_API_KEY" --base-url "$OPENROUTER_BASE_URL"

Each launcher starts a local proxy, points the agent at it, and shuts the proxy down when the agent exits. Use --model for single-model passthrough or --routing-profiles for a route bundle:

switchyard launch claude --model openai/gpt-4o-mini --base-url https://openrouter.ai/api/v1       # single-model passthrough
switchyard --routing-profiles routes.yaml -- launch claude                                        # route bundle

Bedrock-backed profile caveat (Claude Code + MCP): Bedrock enforces a 64-character toolSpec.name cap. Claude Code's MCP bridge can auto-inject longer tool names, producing BedrockException 400s on tool-bearing requests. If you use a Bedrock-backed route and hit this, swap to an OpenAI-compatible model with --model openai/gpt-4o or a routing-profile YAML.

See Agent Launchers for supported harness versions, model requirements, troubleshooting, and Claude Code /model picker aliasing.

2. Run a standalone Python server

Define one or more client-facing routes in YAML. A complete OpenRouter-backed random-routing bundle looks like this:

defaults:
  api_key: ${OPENROUTER_API_KEY}
  base_url: https://openrouter.ai/api/v1
  format: openai

routes:
  smart:
    type: random_routing
    strong:
      model: openai/gpt-4o
    weak:
      model: openai/gpt-4o-mini
    strong_probability: 0.3
    fallback_target_on_evict: strong

Serve it as a proxy. The route name is exposed as a model ID, and clients select it through the request's model field:

switchyard --routing-profiles routes.yaml -- serve --port 4000
curl http://localhost:4000/v1/models
curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "smart", "messages": [{"role": "user", "content": "hi"}]}'

switchyard --routing-profiles routes.yaml -- launch claude
switchyard --routing-profiles routes.yaml -- configure --target provider \
  --provider openrouter --api-key "$OPENROUTER_API_KEY" \
  --base-url https://openrouter.ai/api/v1 --no-tui --no-model-discovery

Non-interactive configure does not read provider credentials from the routing bundle; pass --api-key explicitly when persisting the bundle for CI.

The Rust switchyard-server binary has a separate explicit TOML schema for LLM clients, targets, and libsy algorithms. See crates/switchyard-server/README.md.

For profile selection and full configuration examples, start with Routing Overview, then open the strategy-specific page:

• Random Routing
• LLM Classifier Routing
• Stage-Router Routing

For multi-turn classifier sessions, see Session Affinity (Sticky Routing).

3. Use as a Python library

import asyncio

from switchyard import ChatRequest, PassthroughProfileConfig, ProfileSwitchyard

switchyard = ProfileSwitchyard(PassthroughProfileConfig(
    api_key="sk-...",
    base_url="https://api.openai.com/v1",
).build())

async def main():
    request = ChatRequest.openai_chat({
        "model": "gpt-4o",
        "messages": [{"role": "user", "content": "What is 2+2?"}],
    })
    response = await switchyard.call(request)
    # call() returns a JSON-compatible dict in the OpenAI Chat Completions shape.
    print(response["choices"][0]["message"]["content"])

asyncio.run(main())

Architecture

Switchyard sits between your client applications and one or more LLM backends:

flowchart LR
    clients["Clients"]
    switchyard["Switchyard<br/>routing · translation · fallback"]
    backends["Model backends"]

    clients -->|"OpenAI / Anthropic API"| switchyard
    switchyard -->|"provider-native format"| backends

Loading

Clients keep their native OpenAI or Anthropic API format. Switchyard picks a configured backend, forwards the request in that backend's own format, and translates the response back into the shape the client expects. See Architecture for the system context and the full request flow.

Installation Options

Install from PyPI:

pip install nemo-switchyard

Optional extras:

pip install "nemo-switchyard[server]"   # FastAPI / Uvicorn HTTP endpoints
pip install "nemo-switchyard[cli]"      # Interactive CLI launchers (Claude / Codex)
pip install "nemo-switchyard[all]"      # Server, CLI, tracing, intake, and affinity-redis extras

See Installation for a full breakdown of what each extra adds.

Documentation

• Getting Started: step-by-step setup, first request, troubleshooting
• Known Issues: known issues in 0.1.0
• Agent Launchers: Claude Code, Codex, and OpenClaw launcher behavior
• Cli Reference: canonical reference for every switchyard subcommand and flag
• Architecture: system context and end-to-end request flow
• Routing Algorithms: signal-driven weak/strong stage-router routing: picker layers, signal dimensions, and calibration data.
• Contributing: dev setup, testing, CI gates, PR process
• Development: project structure, benchmarks, conventions
• Agents: full design philosophy and architectural patterns

Supported Providers

• OpenAI: Chat Completions API
• Anthropic: Claude Messages API
• OpenAI Responses API: structured output / reasoning
• OpenAI-compatible APIs: vLLM, Ollama, Azure, etc. (anything with /v1/chat/completions)

Requirements

• Python 3.12+
• macOS, Linux, or Windows
• API keys for your chosen backend (OpenAI, Anthropic, etc.)
• Linux x86_64 wheels require an x86-64-v3 / AVX2-class CPU (post 2013).
• Linux aarch64 wheels require a Neoverse N1-class CPU (post 2020).

Community

• Issues: GitHub Issues
• Code of Conduct: Code of Conduct
• Contributing: Contributing

License

Apache 2.0 License. Copyright NVIDIA Corporation.