test: add agent evals

[angeloashmore]

Resolves:

Description

Adds an eval suite that measures how well AI agents (like Claude Code) use the CLI.

Each eval gives an agent a task in a temp project. The agent runs the real CLI against a real repo, and the test checks the outcome. A few notes on the design:

• Evals are plain Vitest tests. They reuse the existing e2e fixtures and grade results with normal asserts, plus an LLM judge for criteria that need judgement (for example, "are these field types sensible?").
• Each eval runs several trials. We track pass rates per run instead of a single pass/fail, since agent behavior varies.
• Evals are measurements, not gates. Evals the agent fails today stay enabled and score 0% until the CLI improves; an eval is only skipped when the CLI cannot pass it (missing feature).
• Each eval gets its own throwaway repository, so evals run concurrently without interfering and agents are free to push. Global teardown sweeps repositories created by agents that the harness cannot name in advance (e.g. prismic init).
• Results are a single evals/results.json the reporter overwrites each run: trials grouped by eval, sorted by name, with each trial's pass/fail, cost, duration, and the prismic commands it ran. node --run evals:report (a jq one-liner) prints the latest run.
• Run history is deliberately delegated to git: every committed run is a version of results.json, so git acts as the time-aware database. git log -- evals/results.json lists past runs, git show <rev>:evals/results.json reads any of them, and an analysis agent can walk those versions to compare pass rates, cost, and command usage across CLI commits — no run ids, timestamps, or history tooling in the harness itself.
• Iterating is safe by default: filtered runs (node --run evals -- sync-models, -t "adds a field", it.only) write only the gitignored evals/results.local.json, never results.json — the reporter detects them, so a committed snapshot is always a full run. Agents can diff results.local.json against results.json to compare a working-tree run with the last recorded one. The one convention: permanently disabled evals use it.todo, since vitest reports it.skip identically to a filter.
• Evals are opt-in (node --run evals) and refuse to run outside a container or CI, because the agent runs with permissions bypassed.
• A full run is 33 evals × 3 trials: roughly $31 in API-equivalent cost and 10 minutes.

Checklist

• A comprehensive Linear ticket, providing sufficient context and details to facilitate the review of the PR, is linked to the PR.
• If my changes require tests, I added them.
• If my changes affect backward compatibility, it has been discussed.
• If my changes require an update to the CONTRIBUTING.md guide, I updated it.

Preview

node --run evals runs the suite as normal Vitest tests and writes evals/results.json. The file is stored on a single line so each recorded run diffs as one changed line; formatted excerpt:

{
	"model": "claude-sonnet-5",
	"evals": {
		"adds a field": [
			{
				"pass": true,
				"costUsd": 0.35,
				"durationS": 20,
				"calls": [
					"field add --help",
					"field add rich-text excerpt --to-type article --label Excerpt"
				]
			}
		]
	}
}

node --run evals:report prints the latest run, one line per eval (excerpt):

3/3  20s   adds a field
2/3  29s   commits and pushes local model changes
0/3  35s   models a title as single-heading rich text and a social media handle as key text
3/3  216s  models and wires a landing page end to end

How to QA ¹

Inside a disposable container with .env.test.local available:

PRISMIC_ALLOW_EVALS=true node --run evals
node --run evals:report
git diff evals/results.json

🤖 Generated with Claude Code

---

Note

Medium Risk
Evals run a real agent with permission bypass against live Prismic accounts and billable Anthropic usage; risk is confined to isolated/CI runs, with a small CLI change to telemetry env overrides.

Overview
Adds an opt-in Vitest eval project that drives the real prismic CLI through the Claude Agent SDK (with permissions bypassed) against disposable repos and temp Next.js projects, so agent behavior can be measured over multiple trials instead of gated as unit tests.

The harness in evals/it.ts wires the built CLI into each project, appends the published Prismic skill, records every prismic invocation, and exposes matchers like toHaveRun and LLM-based toSatisfyJudge. Scenarios live in new *.eval.ts files (modeling, routes, repo setup, sync/push, init, safety checks, slice components, etc.). A custom reporter writes evals/results.json on full runs and evals/results.local.json on filtered runs, with an evals:report script to summarize pass rates, cost, and duration.

Tooling changes: Vitest is split into tests vs evals projects; node --run evals is guarded by PRISMIC_ALLOW_EVALS=true. E2E fixtures gain a generic exec helper; .env.test.example documents ANTHROPIC_API_KEY. Telemetry/Sentry in src/index.ts now honor explicit PRISMIC_*_ENABLED env overrides (used to quiet eval runs). A baseline evals/results.json snapshot is committed for git-based history.

^{Reviewed by Cursor Bugbot for commit 70b0523. Bugbot is set up for automated code reviews on this repo. Configure here.}

Footnotes

1. Please use these labels when submitting a review:
   ❓ #ask: Ask a question.
   💡 #idea: Suggest an idea.
   ⚠️ #issue: Strongly suggest a change.
   🎉 #nice: Share a compliment. ↩

[lihbr]

Look good to me, I like what you did with Vite and using a custom reporter, the failing evals are also really insightful :)

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit c5fdf22. Configure here.}