[UX] Improved failure handling for rollout collection

[sephmard]

Background

PR #1799 fixes an immediate inconsistency — gym eval run was including failed rollouts in aggregate metrics while gym eval aggregate was not, causing the two commands to disagree. That fix is minimal and correct, but it surfaces a broader set of problems with how Gym handles failures during rollout collection that need to be addressed together.

Problems

No consistent contract for failure handling across agents

Only pinchbench and stirrup_agent emit failed rollouts to *_failures.jsonl instead of crashing. All other agents, including simple_agent (used by most benchmarks), crash the entire run when an unexpected failure occurs. There is no framework-level mechanism to catch failures gracefully — each agent handles (or doesn't handle) failures independently.

Inconsistent destination for failed samples

There is no agreed contract for where a failed rollout goes. *_failures.jsonl exists as a sidecar but is only used by two agents. Judge failures (tracked separately) currently land in output.jsonl, not *_failures.jsonl. The result is that failures end up in different places depending on which agent and which failure mode, making them hard to inspect, resume from, or reason about consistently.

Runs crash instead of completing with partial results

When a rollout fails today, the entire run stops. No progress is saved for the remaining tasks. This means transient infrastructure failures (Ray crashes, timeouts) cause total loss of an eval run rather than recoverable partial results.

Metrics are reported even when the run is incomplete

When failures do occur silently (e.g. via the zero-reward path), aggregate metrics are computed and reported over an incomplete result set without any signal to the user that the numbers are based on partial data.

Intended behavior

• When a rollout fails, don't crash the run — continue collecting remaining rollouts and save the failure to *_failures.jsonl
• Only compute and report aggregate metrics for benchmarks where all rollouts completed successfully
• Exit with a non-zero status code when any failures occurred, so users are aware and CI catches it
• Failed runs are resumable via --resume without re-running successful rollouts
• The failure contract is enforced at the framework level, not per-agent — a non-200 response from any server is treated as a failure and routed to *_failures.jsonl automatically

Relation to #1799

#1799 is the minimal fix — align what gym eval run aggregates with what gym eval aggregate reads. It should merge as-is. This issue tracks the follow-on work to make failure handling robust across the framework. Target: 0.6.

Open questions

• Is the right failure contract a non-200 HTTP status code → *_failures.jsonl, or does it need to be more explicit?
• Should judge failures (currently in output.jsonl) move to *_failures.jsonl to be consistent?
• Should metric reporting fail the whole benchmark if any task has incomplete rollouts, or only if the failure rate exceeds a threshold?

Acceptance criteria

• Framework-level failure handling routes non-successful rollouts to *_failures.jsonl without per-agent changes
• gym eval run does not crash on rollout failure — completes the remaining tasks and exits with a non-zero code
• Aggregate metrics are only computed when all rollouts for a benchmark completed successfully
• Failed runs are resumable without re-running successful rollouts
• Failure handling contract is documented