fix(live): keep trace git metadata and late spans visible

[RitwijParmar]

References #1099 and #1006.

This fixes two live-trace edge cases that come from treating the root span as the whole trace lifecycle.

What changed

• Promote traceroot.git.repo and traceroot.git.ref from child spans into the trace record as soon as they arrive, instead of waiting for the root span. The root span still wins if it carries a value later.
• Keep the live SSE stream open for a short quiet window after a root-complete signal. This lets late descendant span batches arrive before the frontend receives trace_complete and closes the stream.
• Preserve the subscribe-before-ClickHouse-check invariant so concurrent Redis span events are still not missed.
• Add regression coverage for child-before-root git metadata, root-over-child precedence, root-without-git fallback, already-complete quiet completion, and late spans arriving after trace_complete.

Why this matters

For live agent runs, the trace can be useful before the root span lands or before all descendants have arrived. This makes repo/ref visible earlier for debugging and prevents the UI from closing the stream while the trace is still growing.

Validation

• uv run ruff check backend/rest/routers/live.py backend/worker/otel_transform.py tests/rest/test_live_router.py tests/worker/test_otel_transform.py
• uv run python -m py_compile backend/rest/routers/live.py backend/worker/otel_transform.py tests/rest/test_live_router.py tests/worker/test_otel_transform.py
• uv run pytest tests/rest/test_live_router.py tests/worker/test_otel_transform.py -q passed, 98 tests
• git diff --check

---

Summary by cubic

Improve live trace reliability by surfacing git repo/ref early and holding the SSE stream open for a short quiet window after completion. The window is anchored to the latest of the root’s end time and the last ingest activity so late spans are delivered while old completed traces close immediately.

• Bug Fixes
  • Promote traceroot.git.repo and traceroot.git.ref from child spans into the trace as they arrive; root values overwrite, and child values persist if the root has none.
  • Emit trace_complete after a quiet window triggered by ClickHouse root completion or a Redis trace_complete; reset on late spans or redundant trace_complete; anchor the deadline to max(root end, last ingest) and close immediately if it’s already past; subscribe to Redis before the ClickHouse check and forward spans received during the check/window before closing; at the hard ceiling emit trace_complete if the root completed, otherwise stream_timeout; the window length is settings.trace_complete_quiet_seconds (default 10s, above the SDK flush interval).

^{Written for commit ce0261f. Summary will update on new commits.}

[cubic-dev-ai]

No issues found across 4 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

Architecture diagram

sequenceDiagram
    participant Client as Frontend Client
    participant SSE as Live SSE Stream (live.py)
    participant Redis as Redis PubSub
    participant CH as ClickHouse
    participant Worker as Background Worker (otel_transform.py)

    Note over Client,Worker: Live Trace SSE Flow - Happy Path

    Client->>SSE: GET /live/{trace_id}
    SSE->>Redis: subscribe(trace_channel)
    Note over SSE,Redis: NEW: Subscribe before ClickHouse check
    
    SSE->>CH: _is_trace_complete_in_clickhouse()
    alt Already complete in ClickHouse
        CH-->>SSE: root span with end_time found
        SSE->>SSE: NEW: start quiet window (TRACE_COMPLETE_QUIET_SECONDS)
        loop Quiet window active
            SSE->>Redis: poll for messages
            alt Late span batch arrives
                Redis-->>SSE: {"type": "spans", ...}
                SSE->>SSE: NEW: reset quiet window deadline
                SSE-->>Client: event: spans / data...
            else Quiet window expires
                SSE-->>Client: event: trace_complete
            end
        end
    else Not yet complete
        CH-->>SSE: no root span end_time
        SSE->>SSE: stream normally, no quiet deadline
    end

    Note over Worker,CH: Span Processing (concurrent)

    Worker->>Worker: transform_otel_to_clickhouse()
    Note over Worker: NEW: Extract git ref/repo from child spans
    Worker->>CH: write spans (including child git metadata)
    Worker->>Redis: publish(trace_channel, {"type": "spans", ...})
    
    alt Root span arrives later
        Worker->>Worker: NEW: Root git values override child values
        Worker->>Redis: publish(trace_channel, {"type": "trace_complete"})
    end

    Note over SSE,Redis: Late span delivery after root complete

    SSE->>Redis: poll (quiet window still active)
    Redis-->>SSE: {"type": "spans", ...}
    SSE->>SSE: NEW: reset quiet window
    SSE-->>Client: event: spans / data...

    Note over SSE,Client: Completion Event
    SSE->>SSE: quiet window fully expired
    SSE-->>Client: event: trace_complete
    SSE->>Redis: unsubscribe & close
    Client->>Client: close SSE stream

    Note over SSE,Redis: Edge Case: Redundant trace_complete
    alt Redis sends redundant trace_complete
        Redis-->>SSE: {"type": "trace_complete"}
        SSE->>SSE: NEW: reset quiet window only (no duplicate emit)
    end

Loading

_{Re-trigger cubic}

[dark-sorceror]

I also noticed you had a similar PR open for git half (#1120). Looks identical, so let's keep this one and close the other when this is merged.

[dark-sorceror]

@cubic-dev-ai review

[cubic-dev-ai]

@cubic-dev-ai review

@dark-sorceror I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

No issues found across 5 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

Architecture diagram

sequenceDiagram
    participant Client as Frontend Client
    participant SSE as Live Stream Endpoint
    participant Redis as Redis PubSub
    participant CH as ClickHouse
    participant Worker as OTel Transform Worker

    Note over Worker,CH: Background span ingestion (otel_transform.py)

    Worker->>CH: Write span batch
    alt First span for trace is a child with git metadata
        Worker->>Worker: NEW: Promote traceroot.git.repo/ref to trace record immediately
        Worker-->>CH: Trace record includes git_repo, git_ref (child values)
    else Root span arrives later
        Worker->>CH: Update trace record (root values overwrite if present)
    end

    Note over Client,SSE: Live SSE stream (live.py)

    Client->>SSE: GET /live (subscribe)
    SSE->>Redis: Subscribe to trace channel
    Note over SSE,CH: Ensure subscribe before ClickHouse check (unchanged)

    SSE->>CH: Query root_completion_time for trace
    alt Root end_time exists (already has completed root)
        CH-->>SSE: root_end_time
        SSE->>SSE: NEW: Compute quiet deadline anchored to root end time
        alt Deadline already expired (old trace)
            SSE-->>Client: event: trace_complete (immediate)
        else Deadline in future
            SSE->>Redis: listen for messages (timeout = quiet_remaining or heartbeat)
            loop Until deadline or disconnect
                Redis-->>SSE: Optional late spans
                SSE-->>Client: event: spans (forwarded)
                alt Span received
                    SSE->>SSE: NEW: Reset quiet deadline
                end
                alt trace_complete from Redis (redundant)
                    SSE->>SSE: NEW: Reset quiet deadline (no output)
                end
                opt Heartbeat timeout
                    SSE-->>Client: : heartbeat
                end
            end
            SSE-->>Client: event: trace_complete
        end
    else No root end_time (trace still live)
        CH-->>SSE: null
        SSE->>SSE: No quiet deadline set
        loop Until hard ceiling (MAX_STREAM_SECONDS) or disconnect
            Redis-->>SSE: span event
            SSE-->>Client: event: spans
            alt trace_complete from Redis
                SSE->>SSE: NEW: Set quiet deadline (now + TRACE_COMPLETE_QUIET_SECONDS)
                Note over SSE,Client: Continue listening for late spans
            end
            opt Heartbeat
                SSE-->>Client: : heartbeat
            end
        end
        alt Quiet deadline expired (root completed earlier)
            SSE-->>Client: event: trace_complete
        else Timeout before any completion
            SSE-->>Client: event: stream_timeout
        end
    end

    Note over SSE,Client: Frontend closes stream after trace_complete

Loading

_{Re-trigger cubic}

[dark-sorceror]

LGTM