feat: add switchyard_model server for router-backed evals#2141
Open
linj-glitch wants to merge 6 commits into
Open
feat: add switchyard_model server for router-backed evals#2141linj-glitch wants to merge 6 commits into
linj-glitch wants to merge 6 commits into
Conversation
Serves Gym model calls through a Switchyard routing proxy, so any benchmark
Gym already supports can be evaluated against a router without touching
harness code:
gym eval run --benchmark <name> --model-type switchyard_model
Switchyard is integrated at the model layer rather than the agent layer. The
existing Switchyard work in Gym (#2026 token capture, #2080 OpenCode) wires
one harness at a time under responses_api_agents/swe_agents; putting the
router behind a Responses API model instead makes it available to every
benchmark at once, which is what evaluating the router itself requires.
Two modes. Attach (default) points at a proxy run outside Gym, which keeps
proxy lifecycle and image pinning with the eval harness so a run can be
compared against a scaled-evals run of the same Switchyard commit. Launch
starts the proxy from a routing profile and shuts it down at exit, for a
single-command run.
Gym's rollout-attempt id is forwarded as Switchyard's opaque session id, so
proxy-side routing decisions and costs join back to the rollout that produced
them. The correlation prefix is stripped by Gym's capture middleware before
handlers run, so a thin outer middleware republishes the id for the request.
The nemo-switchyard dependency is scoped to this package rather than Gym's
core dependencies: only runs that route through Switchyard pay for it, and
attach mode needs no dependency at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Lin Jia <linj@nvidia.com>
This server never imports Switchyard. It speaks OpenAI-compatible HTTP to the proxy, and launch_proxy mode shells out to the `switchyard` CLI, which only needs the executable on PATH. The dependency bought nothing at import time and made the package unresolvable, since the published nemo-switchyard 0.1.0 pins openai>=2.34.0 against Gym's openai<=2.7.2. Leaving the install to whoever wants launch mode also lets an eval pin the exact Switchyard build its numbers are reported against, rather than inheriting Gym's. start_proxy now checks PATH up front and explains how to fix a missing executable, instead of surfacing an opaque FileNotFoundError from Popen. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Lin Jia <linj@nvidia.com>
linj-glitch
marked this pull request as ready for review
July 25, 2026 00:27
The point of opting into switchyard_model is not having to run a proxy. If the user has to launch one anyway, openai_model pointed at it already works and this server barely earns its place. So launching is now the default rather than a flag: set routing_profiles and Gym starts, health-checks and tears down the proxy for you. switchyard_base_url stays as the escape hatch for attaching to a proxy you run yourself. That removes the launch_proxy boolean -- the mode is inferred from whichever of the two you configured, and configuring neither now fails naming both. The spawn also moves from model_post_init into setup_webserver, matching local_vllm_model. Launching on construction meant any path that merely built the model -- config resolution, validation, tests -- would start a proxy; the tests having to stub Popen just to construct an instance was the tell. Regression test added. proxy_host now defaults to the wildcard address, as local_vllm_model does for cross-node reachability, while this server addresses its own child over loopback, since a wildcard is a bind address and not a destination. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Lin Jia <linj@nvidia.com>
Covers the Gym-managed proxy launch, attaching to a proxy you run yourself, that --model names a route rather than a model, rollout correlation via proxy_x_session_id, and the config fields. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Lin Jia <linj@nvidia.com>
|
🌿 Preview your docs: https://nvidia-preview-feat-switchyard-model-server.docs.buildwithfern.com/nemo/gym Here are the markdown pages you've updated: |
Gym-managed and self-managed proxies are both first-class, and the mode is inferred from whichever field is set. Setting both was resolved silently in favour of attaching, which meant a routing profile the user supplied looked like it was in effect when it was not -- easy to hit by accident, since both fields default from environment variables. Attaching still wins, but now says so. The docs lead with the choice between the two modes rather than presenting self-managed as an afterthought. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Lin Jia <linj@nvidia.com>
Users had to run pip install themselves before launch mode would work. Gym already creates a virtual environment per server and installs that server's pyproject into it, so declaring the dependency here removes the manual step without putting Switchyard anywhere near Gym's core dependencies. The published nemo-switchyard 0.1.0 declares openai>=2.34.0, which Gym's openai<=2.7.2 cannot satisfy. That floor was never a real requirement -- the suite at v0.1.0 passes identically on 2.7.0 through 2.48.0 -- and it was widened in NVIDIA-NeMo/Switchyard#142, but PyPI releases are immutable so the published artifact keeps the old metadata. A uv override neutralizes it. The override restates Gym's own openai pin rather than expressing a preference of its own, so the resolved openai is whatever Gym would have resolved anyway (2.7.2, verified). Overrides apply to every requirement for a package, not just the offending one, so omitting the ceiling would erase Gym's own and pull in a far newer openai than Gym is tested against. nemo-switchyard is pinned exactly, since the override asserts something about one specific published artifact. Both come out once a release with the widened floor ships. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Lin Jia <linj@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
responses_api_models/switchyard_model/— a Gym model server that serves model calls through a Switchyard routing proxy, and runs that proxy for you.Why the model layer
Switchyard decides, per request, which upstream model carries the work. The existing Switchyard integrations in Gym — #2026 (token capture) and #2080 (OpenCode) — both live under
responses_api_agents/swe_agents/and wire one harness at a time. That is the right shape for RL trace capture, but evaluating the router itself means running many benchmarks through it. Putting Switchyard behind a Responses API model makes it available to every benchmark Gym already supports without touching harness code.The motivating use case is running evals with Switchyard as the router (tracked in the Gym master doc under Switchyard <> Gym integration).
Gym runs the proxy
Set
routing_profilesand the proxy is started with this server, health-checked, and stopped when it exits. The user never manages it. Launching drives theswitchyardCLI, so it needs to be on PATH;start_proxychecks that up front and says so rather than surfacing an opaqueFileNotFoundError.Setting
switchyard_base_urlinstead attaches to a proxy you already run. Two reasons to reach for that: pinning a specific Switchyard build so a Gym run can be compared against a scaled-evals run of the same commit, and sharing one instance across servers — routing strategies that use session or agent affinity are stateful, so replicas each running their own proxy would not route the way a single deployed proxy does.The mode is inferred from whichever you set, rather than a separate flag. Configuring neither fails naming both.
Launching happens in
setup_webserver, matchinglocal_vllm_model— so config resolution, validation and tests never start a proxy as a side effect. There's a regression test for that.Rollout correlation
Gym's rollout-attempt id is forwarded as Switchyard's opaque session id (
proxy_x_session_id), so proxy-side routing decisions and costs can be joined back to the rollout that produced them.Gym's capture middleware strips the
/ng-rollout/<id>prefix before handlers run, so a thin pure-ASGI middleware is installed outside it to republish the id on a ContextVar for the duration of the request. The path is forwarded untouched — stripping stays the capture middleware's job.No dependency on nemo-switchyard
Deliberate. This server never imports Switchyard: it speaks OpenAI-compatible HTTP to the proxy and drives the CLI to launch one. Leaving the install to the user also lets an eval pin the exact Switchyard build its numbers are reported against rather than inheriting Gym's.
Dependency direction stays one-way: Gym knows Switchyard; Switchyard does not know Gym.
Testing
ruff checkandruff formatclean.uv lockon the package resolves.gym list modelsshowsswitchyard_modelas a valid--model-type.Covered: mode inference and its failure case, route-name override (a caller-supplied
modelmust not bypass routing), session-id forwarding on correlated calls and its absence on uncorrelated ones,forward_session_id=false,extra_bodymerge, construction not spawning a proxy, missing-executable guidance, proxy command construction, startup failure, health-poll retry, timeout, and terminate/kill on shutdown.Not covered by tests: launch mode has never been run against a live proxy end to end. The argv construction is unit-tested, but a smoke test against a real routing profile is worth doing before anyone leans on it.
Two things reviewers should weigh in on
ModelCallRecord.model_refnames the Gym model server, not the routed target, so "which model served this call" currently lives only in the captured response body — and payloads are excluded from attached records unlessinclude_payloadsis set. Promoting Switchyard's decision to a first-class record field would make routing attribution directly consumable. Deliberately out of scope here; happy to follow up.harbor_agentoverwritesresponse["model"]with the configured policy model name (app.py:316). Any benchmark running through Harbor will reportswitchyard_modelfor every call regardless of what actually served it, so attribution must be read from capture records rather than the rollout response. Flagging in case that overwrite is not intentional.🤖 Generated with Claude Code