tracecore

Names the root cause of a broken GPU training run — by pattern, with evidence — instead of leaving operators to correlate 12 dashboards by hand.

An OpenTelemetry Collector distribution that classifies distributed-training failures into 13 named patterns: NCCL hangs, HBM ECC storms, PCIe AER, IB link flap, CUDA OOM, checkpointer stalls, loss-spike NaN, silent data corruption, and more. Each detector emits a verdict with a citation trail an on-call engineer can act on.

• Try it: docs/getting-started.md — build, run, see a pattern verdict fire.
• Deploy it: install/kubernetes/tracecore/ — production Helm chart with SLO rules + air-gap bundle.
• Verify it: docs/reproducibility.md — reproducible build, cosign signatures, SLSA v1.0 provenance.

Status

Pre-1.0. Safe on eval clusters and CI replay. Not yet a production drop-in — check CHANGELOG.md for what's stable.

Releases are signed (cosign keyless), come with CycloneDX SBOMs + SLSA v1.0 provenance, and rebuild byte-identically.

Why tracecore vs. plain OpenTelemetry Collector?

Tracecore is an OpenTelemetry Collector — built via the OpenTelemetry Collector Builder (OCB) from upstream + contrib components. What it adds on top:

• Pattern detectors + replay corpus. 13 cross-signal root-cause patterns ship as a processor (patterndetectorprocessor) with synthetic-fixture replay tests. No upstream equivalent.
• NCCL FlightRecorder parsing + cross-rank join. A dedicated receiver (ncclfrreceiver) plus a windowed cohort-join processor (rankjoinprocessor); both will upstream to OTel-contrib once the SIG accepts them.
• Pre-wired training-cluster recipes. OTTL normalization for gen_ai.training.*, k8s.event.hint, gpu.vendor — so upstream receivers (filelogreceiver, journaldreceiver, k8sobjectsreceiver, DCGM/ROCm scrape) emit a stable contract operators alert against.
• Standard operator surface. Single OCB-built binary, upstream /metrics + /healthz at :13133/, alerts ship next to RUNBOOKs.

Use plain OTel Collector when your needs are application-tracing-shaped or you want a broader receiver set than the training stack. Use tracecore when you're running distributed GPU training and want root-cause verdicts.

Quickstart

# Build + validate + run against a minimal config that emits load
# averages to stdout. Portable across linux/darwin/windows.
make build
cat > demo.yaml <<'YAML'
receivers:
  hostmetrics:
    collection_interval: 1s
    scrapers:
      load: {}
exporters:
  debug:
    verbosity: basic
service:
  pipelines:
    metrics:
      receivers: [hostmetrics]
      exporters: [debug]
YAML
./_build/tracecore validate --config=demo.yaml
./_build/tracecore --config=demo.yaml

See a pattern verdict fire. For the training-cluster path — receive real signal, watch a detector emit a verdict — walk through docs/getting-started.md §Walkthrough and then §"Add a real receiver".

Documentation

• Operators — docs/getting-started.md, recipes under docs/integrations/, triage guide docs/failure-modes.md.
• Contributors — CONTRIBUTING.md, PRINCIPLES.md, STYLE.md.
• Supply-chain auditors — docs/reproducibility.md.

Full index: docs/README.md.

License

Apache License 2.0.