perf(contribution-check): cut token/tool overhead per #5558#5576
Conversation
Token-optimization pass on the Contribution Check workflow (ranked #1 by total AIC). All review data is already pre-fetched in `steps:`, so the agent only needs to read three context files and emit a single `add_comment` (or noop) safe-output. Changes: - tools: disable GitHub tools (`github: false`) instead of the previous `gh-proxy`/`pull_requests` config. NOTE: simply *removing* the tools block makes gh-aw auto-inject a read-only GitHub MCP server with a *broader* toolset (context,repos,issues,pull_requests), which is the opposite of the optimization goal. Explicit `github: false` drops the github-mcp-server and cli-proxy containers entirely and prevents the ~5.3 stray GitHub API calls/run the agent was making despite the prompt forbidding them. (lock file shrinks ~150 lines.) Keep `edit:` so the agent can read the pre-fetched files. - strict: false -> true (also required removing the internal `sandbox.mcp.version` key, which strict mode disallows). - max-turns: 5 -> 3 (linear task: read files -> compare -> comment). - Prompt: add an explicit "only add_comment or noop" tool constraint. - Recompiled contribution-check.lock.yml; updated the workflow test's max-turns assertion 5 -> 3. Out of scope: recommendation #1 (all Copilot runs report null token_usage) is a separate, systemic api-proxy telemetry-export investigation, not a per-workflow change. Refs #5558 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
✅ Coverage Check PassedOverall Coverage
📁 Per-file Coverage Changes (1 files)
Coverage comparison generated by |
There was a problem hiding this comment.
Pull request overview
Reduces Copilot token/tool overhead in the Contribution Check agentic workflow by removing GitHub MCP tooling from the agent runtime, tightening execution constraints, and updating the compiled lock + CI test to match.
Changes:
- Lowered the agent turn cap from 5 → 3 and enabled strict: true for tighter runtime behavior.
- Disabled GitHub MCP tools for the agent via
tools.github: falsewhile keepingtools.editso the agent can read the pre-fetched context files. - Tightened the prompt to explicitly limit post-read tool usage to
add_comment(max 1) ornoop, and updated the workflow test + regenerated the lock file accordingly.
Show a summary per file
| File | Description |
|---|---|
| scripts/ci/contribution-check-workflow.test.ts | Updates the workflow guard test to expect max-turns: 3. |
| .github/workflows/contribution-check.md | Applies the workflow-level optimization knobs (turn cap, strict mode, tool disablement) and strengthens the prompt’s tool-use constraints. |
| .github/workflows/contribution-check.lock.yml | Regenerates the compiled lock to reflect the updated workflow configuration and removed GitHub MCP/cli-proxy components. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 3/3 changed files
- Comments generated: 0
- Review effort level: Low
|
✅ Smoke Copilot BYOK AOAI (Entra) completed. Copilot AOAI BYOK (Entra) mode operational. 🔓 |
|
📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅ |
|
✅ Build Test Suite completed successfully! |
|
✅ Smoke Claude passed |
|
📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤 |
|
🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅ |
|
🚀 Security Guard has started processing this pull request |
|
✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟 |
|
✅ Smoke Copilot BYOK AOAI (api-key) completed. Copilot AOAI BYOK (api-key) mode operational. 🔓 |
|
✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓 |
|
🔌 Smoke Services — All services reachable! ✅ |
|
✅ Smoke Gemini completed. All facets verified. 💎 |
|
Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded. |
|
❌ Contribution Check failed. Please review the logs for details. |
Smoke Test: Claude Engine Validation
Overall result: PASS
|
🔥 Smoke Test: Copilot PAT — PASS
Overall: PASS · Auth mode: PAT (COPILOT_GITHUB_TOKEN) cc @lpcox
|
🔬 Smoke Test Results
PR: perf(contribution-check): cut token/tool overhead per #5558 Overall: PASS ✅
|
Smoke Test: Copilot BYOK (Direct) Mode ✅PASS — All smoke tests confirmed. ✅ GitHub MCP connectivity verified (2 recent closed PRs) Running in direct BYOK mode via
|
|
Smoke test summary
Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "registry.npmjs.org"See Network Configuration for more information.
|
🔬 Smoke Test: API Proxy OpenTelemetry Tracing
Summary: All functional scenarios pass. Scenario 5 is a runtime-only check that requires a live container; its absence here is expected.
|
|
@lpcox Running in direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) authenticated via Microsoft Entra Overall: PASS
|
Chroot Smoke Test Results
Overall: ❌ Not all tests passed — Python and Node.js versions differ between host and chroot.
|
🏗️ Build Test Suite Results
Overall: 8/8 ecosystems passed — ✅ PASS
|
Smoke Test Results
Overall status: FAIL Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "localhost"See Network Configuration for more information.
|
|
perf(contribution-check): cut token/tool overhead per #5558
|
Smoke Test Results — Services Connectivity
Overall: FAIL —
|
Implements the actionable recommendations from #5558 to reduce the token/tool overhead of the Contribution Check workflow (ranked #1 by total AIC). All review data is pre-fetched in
steps:, so the agent only needs to read three context files and emit a singleadd_comment(ornoop).Changes
tools.github: false(see note)max-turns5 → 3strictfalse → trueThe issue suggested removing the
tools:block entirely. That backfires on this gh-aw version: with no explicittools.github, gh-aw auto-injects a read-only GitHub MCP server with a broader toolset (context,repos,issues,pull_requests) than the originalgh-proxy/[pull_requests]config — adding tool schemas and agithub-mcp-servercontainer, the opposite of the goal.The correct way to drop GitHub tools is the explicit
github: false(same pattern asdoc-maintainer.md). This removes thegithub-mcp-serverandcli-proxycontainers and eliminates the ~5.3 stray GitHub API calls/run. The compiled lock file shrinks by ~150 lines (86 KB → 79 KB).edit:is kept so the agent can read the pre-fetched files; thesafeoutputsMCP that backsadd_commentis unaffected.strict mode
Enabling
strict: truerequired removing the internalsandbox.mcp.version: "latest"key, which strict mode disallows (it's an internal implementation detail).Verification
gh aw compile contribution-check→ 0 errors, 0 warnings.github-mcp-server,GITHUB_TOOLSETS, or acli-proxycontainer;safeoutputsMCP +add_commentpreserved.scripts/ci/contribution-check-workflow.test.tsupdated (max-turns: 5→3) and passing.Risk note (max-turns)
max-turns: 3maps to the api-proxymaxRuns: 3hard cap. If the agent ever needs a 4th LLM invocation it will receive the terminal 403 (max_runs_exceeded) that surfaces as a misleading "authentication failed" engine error (cf. #5552). Removing GitHub tools frees up the turns previously wasted on strayghcalls, so the linear read→comment task should fit comfortably in 3 — but a live test PR (per the issue's checklist) is the right way to confirm before relying on it.Out of scope
Recommendation #1 (all Copilot runs report
nulltoken_usage) is a separate, systemic api-proxy telemetry-export investigation, not a per-workflow change. Not addressed here.Refs #5558