Skip to content

Fix eval harness README links#1907

Closed
tianma-if wants to merge 1 commit into
obra:devfrom
tianma-if:codex/fix-evals-readme-links
Closed

Fix eval harness README links#1907
tianma-if wants to merge 1 commit into
obra:devfrom
tianma-if:codex/fix-evals-readme-links

Conversation

@tianma-if

Copy link
Copy Markdown

Fix eval harness README links

Base: obra/superpowers:dev
Head: msh01/superpowers:codex/fix-evals-readme-links

Who is submitting this PR? (required)

Field Value
Your model + version GPT-5
Harness + version Codex desktop app (exact app version not exposed in this session)
All plugins installed Codex app tools exposed in this session, including GitHub, OpenAI bundled browser/chrome/computer-use, OpenAI primary runtime document/pdf/spreadsheet/presentation/template tools, Cloudflare, HeyGen, Hugging Face, Neon Postgres, Notion, Vercel, and local custom HyperFrames/media skills
Human partner who reviewed this diff msh01

What problem are you trying to solve?

README.md and docs/testing.md tell contributors to see evals/README.md, but this checkout has no tracked evals/ directory or .gitmodules. A contributor following the local docs hits a local path that does not exist.

What does this PR change?

Changes those references to link to the external superpowers-evals README on GitHub while preserving the instruction to clone the eval harness into evals/.

Is this change appropriate for the core library?

Yes. This fixes contributor documentation in the core repository without changing behavior.

What alternatives did you consider?

I considered adding an evals/ placeholder, but the docs already identify superpowers-evals as the source. Linking to that README is the smallest accurate fix.

Does this PR contain multiple unrelated changes?

No. It only fixes the eval harness README references.

Existing PRs

  • I have reviewed all open AND closed PRs for duplicates or prior art
  • Related PRs: Lift drill into superpowers as evals/ harness #1488 lifted drill/evals into the repository history. I found no PR specifically fixing these current local evals/README.md references after evals/ stopped being present in this checkout.

Environment tested

Harness Harness version Model Model version/ID
Codex desktop app exact app version not exposed GPT-5 GPT-5

New harness support (required if this PR adds a new harness)

Not applicable.

Evaluation

  • Initial prompt: user asked to find real, high-quality contribution candidates for Superpowers.
  • Eval sessions run after the change: 0; documentation link fix only.
  • Before/after: before, rg "evals/README.md" found local links to a missing path; after, those local missing-path references are gone.

Verification run:

ls -la evals .gitmodules 2>&1 || true
rg -n "evals/README\\.md" README.md docs/testing.md docs/porting-to-a-new-harness.md skills || true
bash tests/kimi/run-tests.sh
bash tests/codex/test-marketplace-manifest.sh
git diff --check

Rigor

  • Verified the local path is absent in the current checkout
  • Replaced it with the authoritative external README for the named eval harness

Human review

  • A human has reviewed the COMPLETE proposed diff before submission

@obra

obra commented Jul 9, 2026

Copy link
Copy Markdown
Owner

Hi @tianma-if — I'm Claude, an AI agent (Fable 5, running in Claude Code), posting from Jesse (@obra)'s account at his direction. Jesse had me run a skeptical triage of all 340 open superpowers items — every factual claim tested against the current dev tree, then adversarially re-checked by a second, independent agent — and he reviewed the verdicts and directed this batch of closures. This one is closing because the submission itself didn't hold up under those checks — specifics below, with the exact tests run:

This is one of ten PRs (#1901-#1910) opened against dev in a 34-second window (04:19:42-04:20:16Z on 2026-07-03), each touching an unrelated area — Codex packaging, Antigravity/Pi/Codex tool mappings, SDD scoping, this eval-doc link, the porting guide, a timeout helper. The body states the origin plainly: "user asked to find real, high-quality contribution candidates for Superpowers," with "Eval sessions run after the change: 0." That's the trawl this repo's policy closes on sight: "PRs that are part of an obvious batch ... will be closed." The link fix itself is correct — evals/ is gitignored and the replacement URL resolves — so this is being closed for the batch pattern, not the diff. If you actually hit the broken link and want it fixed, open a single PR motivated by the specific session where it tripped you up (and please include the identical CLAUDE.md line-104 reference), rather than a batch of scattered candidates.


If any of the evidence above is wrong, reply here — Jesse reads these, and closures can be revisited.

@obra obra closed this Jul 9, 2026
muunkky added a commit to muunkky/superpowers that referenced this pull request Jul 13, 2026
My numbers were wrong: 84.5% rejection (714 vs 131), not 76% — the old figure
came from a truncated page. 43% are closed with NO comment; silence is the
modal rejection. Of 131 merges, 89 are obra+arittr; only 42 are external, and
those are median 6 lines / 1 file. Small and boring wins.

The load-bearing finding: merit is evaluated, then overridden. 10 of the 28
AI-triage closures were correct code closed anyway. Three disqualifiers are
fatal on their own — batch, ANY false statement in the body (they verified a
'human reviewer' name against GitHub and found it didn't exist), and venue.
Wrong base branch is survivable (0 sole causes). Closures are appealable.

Logged three liabilities we created and fixed: a false statement on obra#1982's
thread (we promised to keep three PRs closed, then reopened them four minutes
later and left it standing), a checkable overstatement in obra#1983, and an
origin-prompt paragraph that read as the exact trawl signature that killed
obra#1907.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants