Live Tracing Correctness — connected trees, stable ordering, reliable completion, resilient streams

[dark-sorceror]

Summary

Smooth out live SSE traces. While a trace is in flight, the live view often looks broken even though stored data is correct: child spans render disconnected from parents, siblings reorder between updates, git metadata shows up late, the stream can end before late spans arrive, and on timeout/error the UI silently goes stale. Everything snaps correct at completion — which is exactly why users stop trusting the live view.

The trigger is structural: both SDKs export spans only when they end (5s flush / 100-span batches), so leaf spans stream in referencing parents that haven't been exported yet, and the long-lived root span arrives last. Verified against examples/: 28 of 30 example apps (17/19 Python, 11/11 TypeScript entry files with a root wrapper) exhibit this on every run — in an instrumented 20s agent-style run, 16 of 17 spans arrived orphaned and the root appeared only in the final SSE event.

This epic lands the open contributor PRs that fix the data-shape problems, hardens the frontend stream handling, smooths the visual churn from bursty updates, pins it all with regression tests, and closes a set of low-risk efficiency gaps on the same path. No SDK changes are required.

Product framing

The live trace view is a trust-critical surface: it is the first thing a user watches after instrumenting their app. Today it degrades predictably on every agent-style run (long-lived root spans, bursty batch export) and recovers only at completion. This epic makes the in-flight view dependable with the pieces we already have — no protocol or schema changes.

Goals

• Connected trees mid-run: children whose parents haven't arrived render under pending placeholders, never orphaned at the root.
• Stable ordering: same-start siblings keep a deterministic order across live updates and match the post-completion view.
• Reliable completion: the stream doesn't close before late spans arrive; trace-level git metadata appears early.
• Resilient streams: timeout and error recover to a correct view (refetch + resubscribe while the trace is still live); the UI never presents stale data as live.
• Smooth updates: a burst of span events produces one smooth update, not several reflows.
• Efficient live path: per-update work is proportional to new data, not repeated full passes over the whole trace; the ingest path doesn't block the event loop or reconnect per batch.

Non-goals (this epic)

• SDK span-start events or any SDK processor work.
• Backend start-row/upsert implementation; ClickHouse schema changes.
• Removing the final completion refetch (perf(ui): stop full trace re-download on complete & window-focus #1043 stays parked — it remains the correctness safety net).
• Last-Event-ID/resume protocol.

Architecture

SDK (py/ts) BatchSpanProcessor — spans export on END only (5s flush / 100-span batch)
   │ gzip'd OTLP protobuf
   ▼
POST /v1/traces (backend/rest/routers/public/traces.py) ──► S3
   │                                                         │
   │                                     Celery process_s3_traces (backend/worker/ingest_tasks.py)
   │                                         │ 1. write spans to ClickHouse
   │                                         │ 2. publish trace:live:{project}:{trace}
   ▼                                         ▼
GET /projects/{p}/traces/{t}/live (backend/rest/routers/live.py) ◄── Redis pub/sub (no backlog)
   │ SSE: `spans` | `trace_complete` | `stream_timeout` (600s ceiling) | heartbeats
   ▼
use-trace-stream.ts (EventSource) — mergeSpans → enrichSpansWithPending → setQueryData
   ▼
SpanTreeView / SpanTimelineView — enrich again → buildSpanTree / flattenTreeWithMetrics

Clients connecting mid-run get no replay from Redis; everything published before subscribe must come from the base fetch — which strips span metadata (what #1277 repairs).

Plan

Phase 1 — review and land contributor PRs, in this order

1. fix(traces): stabilize sibling span ordering #1171
2. fix(live): keep trace git metadata and late spans visible #1134
3. fix: keep live trace tree connected while parents are pending #1277

Phase 2 — frontend resilience, smoothness, and tests

• Handle stream_timeout and SSE errors in the live trace stream hook #1487, then Cache span metadata parsing and enrich spans once per live update #1488, then Coalesce span-tree re-derivation across SSE event bursts #1489 — same hook/derivation path, land in order.
• Regression tests for the live tracing path #1490 regression tests — after Phase 1 (they pin behaviors the PRs introduce).

Phase 3 — live-path efficiency

• Make buildSpanTree iterative and sort on precomputed timestamps #1491, Stop blocking the event loop in the trace ingest endpoint #1492, Reuse a per-process Redis connection in the ingest worker #1493 — independent of each other and of Phase 2; Stop blocking the event loop in the trace ingest endpoint #1492 and Reuse a per-process Redis connection in the ingest worker #1493 are backend-only and can start immediately.

Sub-issues (7) — dependency order

The (N) labels map to the nodes in the dependency graph below.

• (1) — Handle stream_timeout and SSE errors in the live trace stream hook #1487
• (2) — Cache span metadata parsing and enrich spans once per live update #1488
• (3) — Coalesce span-tree re-derivation across SSE event bursts #1489
• (4) — Regression tests for the live tracing path #1490
• (5) — Make buildSpanTree iterative and sort on precomputed timestamps #1491
• (6) — Stop blocking the event loop in the trace ingest endpoint #1492
• (7) — Reuse a per-process Redis connection in the ingest worker #1493

Dependency graph

Phase 1 (PRs: #1171 ──► #1134 ──► #1277) ──► (4) regression tests
     │
     ├──► (1) ──► (2) ──► (3)   (same hook/derivation path; sequential)
     │
     ├──► (5)                    (pure-function rewrite; parallel-safe)
     │
     └──► (6), (7)               (backend; independent, can start immediately)

• Critical path: Phase 1 PRs → (1) → (2) → (3); everything else hangs off Phase 1 only to avoid rebase churn.
• (6) and (7) touch no frontend files and no PR-contested regions — start them now.
• (1)–(3) all modify use-trace-stream.ts's setQueryData block; sequencing avoids three PRs rewriting the same lines. Re-measure whether (3) is still needed after (2) — (2) removes most of the per-event cost that makes bursts visible.
• (2) and (5) should share one precomputed-timestamp cache rather than build two (cross-referenced in both issues).

Epic acceptance criteria

• Live tree stays connected mid-run (placeholders, not root-pinned orphans).
• Sibling order is stable across live updates and arrival orders, and matches the post-completion view.
• Late spans arrive before the stream closes in the root-ends-first scenario.
• Git metadata appears on the trace before the root span arrives.
• stream_timeout on a still-live trace resumes streaming (refetch + resubscribe); a completed trace shows a final, clearly non-live state.
• Stream state is actually surfaced in the UI, and SSE errors never leave a stale view presented as live.
• A burst of span events produces one smooth update instead of several reflows.
• Per-update client work is bounded: at most one metadata parse per span and one enrichment pass per update.
• The ingest endpoint performs no synchronous network/CPU work on the event loop; the worker publishes without a per-batch reconnect.
• The final refetch agrees with the live-built trace (no snap-change at completion beyond placeholder upgrades).
• Worker publish, live-router timeout/completion paths, the stream hook, and ordering determinism are covered in CI.
• fix(worker): surface git metadata during live traces #1120/fix(worker): populate trace-level git_repo and git_ref from child spans #1106 closed with clear rationale; exactly one git-metadata fix merged; no SDK changes introduced.