Skip to content

docs: add dogfood report for v3.16.0 - #2143

Merged
carlos-alm merged 4 commits into
mainfrom
docs/dogfood-report-v3.16.0
Jul 20, 2026
Merged

docs: add dogfood report for v3.16.0#2143
carlos-alm merged 4 commits into
mainfrom
docs/dogfood-report-v3.16.0

Conversation

@carlos-alm

Copy link
Copy Markdown
Contributor

Summary

Dogfooding report for v3.16.0 — full command sweep, incremental/staleness testing, native-vs-WASM engine comparison, release-specific feature validation, and performance benchmarks. Also includes a few self-improvements to the /dogfood skill itself based on lessons from this session (correct benchmark script paths, registry-isolation guidance, hook-contamination warning, full-edge-diff methodology for catching parity bugs).

Bugs found during this session were filed as #2134-#2140; #2135 and #2136 were fixed in #2141 and #2142.

See generated/dogfood/DOGFOOD_REPORT_v3.16.0.md for full details.

…s from v3.16.0 session

- benchmark scripts are .ts, not .js; note npm run benchmark alias and
  git fetch --tags requirement for correct version labeling
- warn against running registry prune --ttl 0 against the real global
  registry; recommend CODEGRAPH_REGISTRY_PATH isolation instead
- warn about update-graph.sh hook contamination risk from scratch files
  with tracked extensions
- recommend copying .codegraphrc.json aside before editing it in place
  and testing config-driven commands from a different cwd than the target
- recommend full edge-set diffs (not just counts) for incremental rebuild
  verification — this is what caught #2138 in the v3.16.0 session
@greptile-apps

greptile-apps Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds a comprehensive v3.16.0 dogfood report and backports the lessons learned from that session into the /dogfood skill itself. The report covers ~60 command/flag combinations, incremental-rebuild staleness testing, native-vs-WASM engine parity, release-specific feature validation, and partial performance benchmarks, with 7 bugs filed (#2134#2140) and 2 already fixed in #2141/#2142.

Confidence Score: 5/5

Documentation-only change — no executable code is modified; the skill guidance and generated report carry no runtime risk.

Both changed files are Markdown. The SKILL.md improvements are accurate and grounded in real bugs found during the session. The only substantive issue is a severity-label inconsistency for BUG 6 between §9 and §12, which could confuse issue-triage but has no runtime impact.

No files require special attention — both changes are documentation only.

Important Files Changed

Filename Overview
.claude/skills/dogfood/SKILL.md Adds hook-contamination warning, corrects benchmark script paths (.js → .ts), improves edge-set-diff methodology guidance, and strengthens registry-isolation and config-cwd safety notes — all grounded in real bugs found in this session.
generated/dogfood/DOGFOOD_REPORT_v3.16.0.md New dogfood report covering full command sweep, incremental/staleness, engine comparison, benchmarks, and 7 bugs found (#2134#2140); comprehensive and well-structured, with one minor severity-label inconsistency for BUG 6 between §9 and §12.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[SKILL.md updates] --> B[Hook contamination warning]
    A --> C[Benchmark paths .js to .ts]
    A --> D[Edge-set diff methodology]
    A --> E[Registry isolation guidance]
    F[DOGFOOD_REPORT_v3.16.0.md] --> G[Sections 1-3 Setup and Command Sweep]
    F --> H[Section 4 Incremental / Staleness]
    F --> I[Section 5 Native vs WASM Engine Comparison]
    F --> J[Section 6 Release-Specific Feature Tests]
    F --> K[Section 8 Performance Benchmarks]
    F --> L[Section 9 7 Bugs Found]
    L --> M[#2134 hook wrong-repo rebuild]
    L --> N[#2135 audit misleading msg - fixed #2142]
    L --> O[#2136 export --functions ignored - fixed #2141]
    L --> P[#2137 config cwd-sensitivity]
    L --> Q[#2138 incremental edge loss]
    L --> R[#2139 native/WASM parity gap]
    L --> S[#2140 spurious engine warning]
    D -.->|methodology that caught| Q
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[SKILL.md updates] --> B[Hook contamination warning]
    A --> C[Benchmark paths .js to .ts]
    A --> D[Edge-set diff methodology]
    A --> E[Registry isolation guidance]
    F[DOGFOOD_REPORT_v3.16.0.md] --> G[Sections 1-3 Setup and Command Sweep]
    F --> H[Section 4 Incremental / Staleness]
    F --> I[Section 5 Native vs WASM Engine Comparison]
    F --> J[Section 6 Release-Specific Feature Tests]
    F --> K[Section 8 Performance Benchmarks]
    F --> L[Section 9 7 Bugs Found]
    L --> M[#2134 hook wrong-repo rebuild]
    L --> N[#2135 audit misleading msg - fixed #2142]
    L --> O[#2136 export --functions ignored - fixed #2141]
    L --> P[#2137 config cwd-sensitivity]
    L --> Q[#2138 incremental edge loss]
    L --> R[#2139 native/WASM parity gap]
    L --> S[#2140 spurious engine warning]
    D -.->|methodology that caught| Q
Loading

Reviews (3): Last reviewed commit: "docs: explain 741 vs 993 file-count disc..." | Re-trigger Greptile

Comment on lines +249 to +253
- **Fix applied:** none this session (repo-tooling, not product code).

### BUG 2: incremental rebuild loses 10 receiver-dispatch edges via an unrelated file (Medium)
- **Issue:** [#2138](https://github.com/optave/ops-codegraph-tool/issues/2138)
- **PR:** open — too complex for this session (native Rust incremental orchestrator)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Unexplained file-count discrepancy between cold-start build and benchmark

The cold-start build in §2 and the engine-comparison table in §5 consistently report 993 files, but the build benchmark here shows 741 files. A reader consulting these numbers across sections will find them irreconcilable without explanation. If the benchmark script uses its own fixed input set or a different --db target, a brief parenthetical (e.g. "741 files — benchmark uses its own fixtures, not the full source tree") would prevent confusion when future sessions compare figures.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Claude Code

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed — added a parenthetical to the build-benchmark row explaining that scripts/benchmark.ts excludes tests/benchmarks/resolution/fixtures/** (per resolveBenchmarkExcludes() in scripts/lib/bench-config.ts), which is why it sweeps 741 files instead of the 993 used in the cold-start and engine-comparison sections.

…2143)

Build benchmark scripts exclude tests/benchmarks/resolution/fixtures/**
(hand-annotated resolution fixtures), unlike the full-repo cold-start and
engine-comparison scans elsewhere in the report. Addresses Greptile review
feedback flagging the unexplained discrepancy.
@carlos-alm

Copy link
Copy Markdown
Contributor Author

Addressed Greptile's review feedback:

  • Build-benchmark file-count discrepancy (741 vs 993 files): added a parenthetical to the "Full build" row explaining scripts/benchmark.ts excludes tests/benchmarks/resolution/fixtures/** (per resolveBenchmarkExcludes()), unlike the full-repo scans used in the cold-start and engine-comparison sections.
  • Unchecked "General Testing Plan (Any Release)" checklist (flagged as a "Comments Outside Diff" finding, line 374-382): all 8 items were in fact performed this session, so checked them off and annotated each with the report section demonstrating it (§1-§2, §3, §4, §5, §7). This section is a per-session record in the generated report, not a reusable template (the reusable template lives in .claude/skills/dogfood/SKILL.md), so leaving it unchecked was simply an oversight.

@carlos-alm

Copy link
Copy Markdown
Contributor Author

@greptileai

@carlos-alm
carlos-alm merged commit 8147ce1 into main Jul 20, 2026
12 checks passed
@carlos-alm
carlos-alm deleted the docs/dogfood-report-v3.16.0 branch July 20, 2026 17:09
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 20, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant