Capture-doc audit metadata + hub admin maintainer tools (bd-eiku4ymo) - #415
Merged
Conversation
Capture docs gain a top-level automerge `meta` map beside
content/mimeType/hash: kind ("engine-capture"), schemaVersion (1),
createdAt (RFC 3339 UTC), sourcePath (project-relative), engines.
Everything else about a capture is inside the gzipped payload,
invisible to sync-server audits; this envelope is what `hub admin
scan` (Phase B) reads for inventory and its age gate — without
decompressing anything.
- quarto-hub/resource.rs: CaptureDocMeta, create_capture_document
(+ _at test seam for fabricating old captures), read_capture_meta
(every field optional — future/buggy writers stay inspectable),
and CAPTURE_MIME_TYPE moves here as the single source of truth.
- The three capture writers (quarto-preview eager driver +
re-execute, quarto-hub-provider exec server) stamp the envelope;
the previously-duplicated MIME constants become re-exports.
- Written once at creation, never mutated — no CRDT merge concerns.
Legacy docs (no meta) read back as None and stay valid; readers of
`content` are untouched.
- chrono promoted from dev-dep to dep of quarto-hub (RFC 3339
formatting).
Tests: 4 unit tests on the envelope (schema intact, meta roundtrip,
RFC 3339 createdAt, explicit-timestamp seam, legacy None) + the
eager-capture integration test now asserts the recorded doc carries
meta with the right sourcePath/engines. Plan (incl. the reviewed
admin-tool design for Phases B1-B4):
claude-notes/plans/2026-07-24-capture-meta-and-hub-admin-tools.md
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
quarto-hub/src/admin/classify.rs: pure, storage-free classification of stored automerge docs by ROOT shape (project-index, project-set, text-file, engine-capture, binary-file, unknown) and extraction of the liveness-graph edges (index files map + captures sidecar; project-set keys + entry indexDocIds, automerge:-prefix normalized). Unknown is load-bearing: a future schema classifies as Unknown and the collector's allowlist (EngineCapture only) protects it automatically. Roots = index + project-set docs (their inbound pointers live outside the store). 7 unit tests incl. the future-schema and defensive-reference cases. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
admin/scan.rs: enumerate a samod store, load + classify every doc, compute liveness as a worklist closure from the roots (project indexes + project sets), and emit a versioned camelCase manifest (admin/manifest.rs) of orphaned engine-capture docs past the age gate — with per-doc evidence strings, a full by-kind inventory, and a notCollectible section for unreferenced-but-protected docs. Design discovery, recorded in the module docs: whole-store enumeration must NOT go through Storage::load_range([]) — samod's filesystem adapter splays a key's first component across two path segments and load_range rebuilds keys from raw path components, so an empty-prefix listing returns doc ids split in half (samod itself only ever uses per-doc prefixes, where the mapping round-trips). Enumeration is therefore an explicit backend-specific step (list_doc_ids_filesystem mirrors the splay; the adapter's storage-adapter-id file is skipped structurally), while per-doc chunk loading stays on the adapter and is correct on every backend. Caught by the real-store integration test before it could ever mislead a scan. Tests: 6 unit tests on in-memory stores (orphan detection, age gate, --include-unstamped, unknown/binary protection, project-set liveness closure, JSON roundtrip) + a real-store integration test (new tests/integration/ aggregator): docs created through a live HubContext repo, flushed via Repo::stop, scanned offline through TokioFilesystemStorage — exactly the superseded capture is flagged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
admin/collect.rs implements the mutating maintainer tools with every damage-minimization gate from the reviewed design: - collect accepts ONLY a scan manifest (version-checked; data-dir match enforced so a manifest from server A can't authorize collection on server B), re-verifies each candidate against CURRENT storage (exists, still capture-kind, still unreferenced, still past the manifest's own age gate), and QUARANTINES — one rename per doc directory into <data_dir>/trash/<batch>/docs/<id>, chunk SHA-256s recorded first, batch.json embedding the authorizing manifest + skips for the audit trail. Never unlinks. Dry-run by default. - restore moves a batch (or subset) back, refusing per-doc on hash mismatch or when the target dir reappeared (doc re-created). - purge is the only unlink in the tool set: whole trash batches, batch-level, older than the retention window; batches without a readable batch.json have no age evidence and are never eligible. - AdminLock: all three ACQUIRE the server's own hub.lock (exclusive fs2 flock) for their duration — a running server makes them refuse, and they symmetrically block a server starting mid-operation. Integration tests (real HubContext-built store): full lifecycle (dry-run tree-fingerprint inertness, quarantine, project still serves live docs, hash-verified restore, retention-gated purge), re-verification skip when the orphan is re-referenced between scan and collect, and refusals for wrong data dir + held lock. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
hub grows a subcommand layer (serve stays the default — the flat CLI
is unchanged): hub admin scan/collect/restore/purge wiring the
Phase B1-B3 machinery, with negative --older-than-days /
--retention-days as explicit gate-disable escape hatches
(allow_hyphen_values). `quarto hub admin` comes along for free.
Runbook: claude-notes/instructions/hub-storage-hygiene.md (pipeline,
safety properties, peer + ephemeral-data-dir caveats).
E2E (recorded in the plan): store built entirely by shipped binaries
(q2 preview + a real re-execute to orphan a capture), then the hub
binary drove live scan -> offline scan -> dry-run -> quarantine ->
restore -> retention-gated purge; output inspected at each step. The
first attempt ran a stale pre-Phase-A q2 whose captures were
genuinely unstamped — scan correctly protected them ("unstamped
(pre-envelope), excluded by default"), live-validating the
legacy-capture gate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cargo xtask verify --skip-hub-build passes (Rust-only change: quarto-hub, quarto-preview, quarto-hub-provider — hub-client/WASM do not depend on quarto-hub). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
Conflict: both sides created quarto-hub's tests/integration/main.rs aggregator (this branch for the admin tests, main for the sliding-sessions move of auth_bearer + session_auth + support). Resolved as the union, alphabetized. 391 quarto-hub tests pass on the merged tree; clippy clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cscheid
added a commit
that referenced
this pull request
Jul 27, 2026
…surrected deleted content (bd-9fwn1504) (#422) * Investigate bd-9fwn1504: needs_plan drops reconciliation plans on deletion/reorder Root cause of the reconciliation_preserves_structure_full_ast proptest counterexample: the needs_plan optimization (compute.rs) treats an all-KeepBefore alignment as proof the contents are identical, but alignments are per-executed-item — extra original items (deletions) and hash-anywhere reorderings slip through. Apply-side fallbacks then use original content wholesale, resurrecting deleted blocks. Four sites affected: table caption, table cells, custom-node Blocks/Inlines slots. Confirmed with a minimal hand-built repro (caption [a,b] -> [a] keeps 2 blocks). Plan skeleton + evidence committed; seed file left untracked on purpose (it is the fix commit's TDD pin). Discovered bd-fp069xyh: table structural_eq is simplified, hiding the cell-site bug from the property test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * bd-9fwn1504: record design decision — option (b), always store the plan Design settled with Carlos: delete the needs_plan optimization at all 4 sites rather than harden its predicate. Key facts from the consumer audit: the nested plan is fully computed before needs_plan discards it (no compute saved), plans are transient and never serialized in production, and the incremental writer reads only block_alignments + top-level inline_plans, so nested-plan storage cannot affect it. Required cleanup recorded: the apply-side no-plan fallbacks encode the false "no plan means content matched" invariant and must be deleted with the fix. Phase 2 (table structural_eq tightening) deferred to bd-fp069xyh. Unit-test matrix: 4 sites x {deletion, reorder}. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(reconcile): store nested plans unconditionally — plan omission resurrected deleted content (bd-9fwn1504) The needs_plan optimization dropped a nested ReconciliationPlan when every executed-side alignment was KeepBefore, inferring "contents identical". But alignments are one-per-executed-item: the check missed deletions (original has extra items) and reorderings (KeepBefore matches by hash at any position). The apply-side no-plan fallbacks then used original content wholesale, resurrecting deleted blocks/inlines and restoring pre-edit order. Found by CI drawing a failing case of property_tests::reconciliation_preserves_structure_full_ast on PR #415; the counterexample was a table whose executed caption.long lost 3 of 4 blocks and got them all back. Four sites shared the fallacy: table caption.long, table cells, custom-node Slot::Blocks, custom-node Slot::Inlines. Fix (design settled in the plan, option (b)): store the computed plan unconditionally at all 4 compute sites — the plan was already fully computed before needs_plan discarded it, so this costs no compute — and delete the apply-side fallbacks that encoded the false invariant: - cells: no plan now means "no original cell at this position" (row/column growth); executed wins, never original content - caption: no plan means at least one side had no long caption; executed's caption is authoritative - custom slots: original is kept only for single-node slots (Block/ Inline), where plan omission IS eq-verified identity; sequence slots (Blocks/Inlines) always carry a plan, so the fallback prefers executed For provably identical content the stored identity plan moves the original nodes exactly like the old fallback did, so unchanged content still keeps original source infos (asserted by the new tests). Tests (TDD, all 8 verified failing first with the predicted modes): 4 sites x {deletion, reorder} in lib.rs "Plan-omission soundness tests". The CI seed is committed as proptest-regressions/lib.txt (the regression pin; it deterministically reproduced the failure before the fix). Property tests also pass at PROPTEST_CASES=5000. Deferred: tightening the simplified table structural_eq (which hides cell-content divergence from the property test) is tracked separately as bd-fp069xyh. Plan: claude-notes/plans/2026-07-27-reconcile-proptest-counterexample.md Workspace: cargo nextest run --workspace -> 10508 passed, 0 regressions. Full cargo xtask verify (WASM closure) green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * plan: record commit/PR state for bd-9fwn1504 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two deliverables (design reviewed in
claude-notes/plans/2026-07-24-capture-meta-and-hub-admin-tools.md):Part A — uncompressed audit metadata on capture docs. Engine-capture binary docs gain a top-level automerge
metamap (kind,schemaVersion,createdAtRFC 3339,sourcePath,engines) besidecontent/mimeType/hash— everything else about a capture is inside the gzipped payload, invisible to sync-server audits. All three capture writers stamp it;CAPTURE_MIME_TYPEconsolidates into quarto-hub as the single source of truth. Legacy docs (nometa) stay valid; readers ofcontentuntouched; written once, never mutated (no CRDT merge concerns).Part B — sync-server maintainer tools (
hub admin …). Motivated by the orphaned-capture lifecycle: every re-execute mints a new capture doc and repoints the sidecar, orphaning the old one forever (and #410 made captures big). The safety model is a pipeline of independent gates — a mistake must survive all of them before bytes are gone:scan(read-only, live-safe): classifies every doc by ROOT shape (project-index / project-set / text-file / binary-file / engine-capture / unknown), computes liveness as a closure from the roots, applies an age gate onmeta.createdAt, and emits a versioned camelCase manifest with per-doc evidence + a full inventory. Unknown shapes and unstamped captures are structurally protected.collect: accepts only a scan manifest (version- and data-dir-checked), re-verifies every candidate against current storage, and quarantines (renames into<data-dir>/trash/<batch>/with per-chunk SHA-256s and the authorizing manifest embedded inbatch.json). Never unlinks. Dry-run by default.restore: hash-verified undo of a batch or named docs.purge: the only unlink in the tool set — whole trash batches past a retention window, dry-run by default.hub.lock(exclusive flock) for their duration: they refuse under a live server, and a server can't start mid-operation. Serve remains the default command — the flathubCLI is unchanged;quarto hub admincomes for free.Operator runbook:
claude-notes/instructions/hub-storage-hygiene.md(pipeline, safety properties, peer/ephemeral-dir caveats).Implementation note worth review: whole-store enumeration deliberately does not use
Storage::load_range([])— samod's filesystem adapter splays a key's first component across two path segments andload_rangerebuilds keys from raw path components, so an empty-prefix listing returns doc ids split in half. Enumeration is an explicit backend-specific step (list_doc_ids_filesystem); per-doc chunk loads stay on the adapter (correct on every backend). Caught by the real-store integration test.Test plan
q2 preview+ a real/api/preview/re-executeto orphan a capture), then thehubbinary drove live scan → offline scan → dry-run → quarantine → restore → purge; output inspected at each step. A first attempt with a stale pre-Part-Aq2produced genuinely unstamped captures — scan correctly refused to collect them, live-validating the legacy-capture gate.cargo xtask verify --skip-hub-buildpasses (Rust-only change; hub-client/WASM don't depend on quarto-hub).Related: bd-qbhp2cvv (#410). Follow-up material surfaced by scan's
notCollectiblereporting (unreferenced non-capture docs) is deferred per design review.🤖 Generated with Claude Code