Add docs for annotating agent runs by strawgate · Pull Request #2085 · pydantic/logfire

strawgate · 2026-07-18T02:15:12Z

What this adds

A new documentation page for the Annotations feature at docs/reference/advanced/run-annotations/index.md, registered in the nav under Evaluate (next to Evals and Datasets).

Annotations let you review your AI agents' output and label each run with a verdict (pass / neutral / fail), a category, the answer it should have given, a comment, and tags, then triage them across agents and export the results as eval cases.

Page contents

Orientation: what annotations are, why you'd use them, and when to reach for live evals instead.
Annotate a run: numbered UI steps from the agent Runs tab, with keyboard shortcuts.
What you can record: the verdict, category, expected output, comment, and tags fields.
Review across agents: the project-wide Annotations page (Agents grid + recent-annotations feed, with verdict/annotator filters).
Export annotations as eval cases: the JSON Lines export and how it feeds a dataset.
Verify and Troubleshooting sections.

The feature is marked Beta using the standard !!! note "Beta: feedback welcome" admonition. Written to the repo's documentation style guide (sentence-case headings, terms glossed on first use, no em-dashes, no hype words) and grounded in the current UI. Drafted with AI assistance and human-reviewed against the shipped product.

This documents the same feature the platform-side rollout releases from behind its feature flag; the run-annotations walkthrough already points its public_docs frontmatter at this path.

🤖 Generated with Claude Code

https://claude.ai/code/session_01A8uqUP5UYj7vEmofWe4mHW

Generated by Claude Code

Document the Annotations feature: what it is, when to use it (and when to reach for live evals instead), how to annotate a run from the agent Runs tab, the fields you can record, the project-wide Annotations page, and exporting annotations as eval cases. Includes Verify and Troubleshooting sections and marks the feature Beta. Registered in the nav under Evaluate, next to Evals and Datasets. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A8uqUP5UYj7vEmofWe4mHW

coderabbitai · 2026-07-18T02:17:50Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds a new documentation page for beta Logfire Annotations, covering annotation workflows, fields, review views, JSON Lines exports, verification, troubleshooting, and next steps. Adds the page to the “Evaluate: Live Monitoring” navigation under “Annotate Agent Runs.”

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly matches the main change: adding documentation for annotating agent runs.
Description check	✅ Passed	The description directly explains the new Annotations documentation page and its nav entry.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch claude/annotations-feature-flag-rollout-9g08iu

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/reference/advanced/run-annotations/index.md`:
- Line 9: Update the Slack community link in the annotations documentation to
use the source-relative path ../../../join-slack/ instead of the hard-coded
absolute URL, while leaving the surrounding message and email link unchanged.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 84a2c83d-d237-4227-b408-155fe788e9d9

📥 Commits

Reviewing files that changed from the base of the PR and between db9859a and 321ef32.

📒 Files selected for processing (2)

docs/reference/advanced/run-annotations/index.md
mkdocs.yml

🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

pydantic/logfire (manual)
pydantic/pydantic-ai (manual)
pydantic/platform (auto-detected) → reviewed against branch claude/annotations-feature-flag-rollout-9g08iu instead of the default branch
pydantic/pydantic (auto-detected)

Replace the hard-coded absolute URL (which carries the deployment-specific `/docs` prefix) with a source-relative link, matching the repo docs convention and the existing `faq.md` usage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A8uqUP5UYj7vEmofWe4mHW

Review pass against the shipped UI found three inaccuracies: - Verdict is not a required field: an annotation can be saved with just a comment or tag, so it's described as the headline label with the rest as optional context. - Category and expected output are verdict-conditional (category for neutral/fail, expected output for fail); note when each appears. - Export honors the runs table's verdict/category/tag filters, not the annotator chip; the export and troubleshooting text said otherwise. Also fixed the "no agents" troubleshooting: the 7-day grid window is fixed, so point users to the Agents page for older agents instead of "widening" it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A8uqUP5UYj7vEmofWe4mHW

Style guide flags "just" as filler; use "only". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A8uqUP5UYj7vEmofWe4mHW

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/reference/advanced/run-annotations/index.md`:
- Around line 31-32: Update the annotation workflow step before the optional
fields to state that selecting a verdict is optional. Explicitly mention that
users can save comments or tags without selecting a verdict, while preserving
the existing save and advance instructions.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 85033cf9-a38e-4c3c-852d-040724379177

📥 Commits

Reviewing files that changed from the base of the PR and between e839965 and 7ff7a93.

📒 Files selected for processing (2)

docs/reference/advanced/run-annotations/index.md
mkdocs.yml

🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

pydantic/logfire (manual)
pydantic/pydantic-ai (manual)
pydantic/platform (auto-detected)
pydantic/pydantic (auto-detected)

🚧 Files skipped from review as they are similar to previous changes (1)

mkdocs.yml

coderabbitai · 2026-07-23T23:28:29Z

+4. Add any of the optional fields below. Which ones appear depends on the verdict you picked.
+5. Save. Press `cmd+enter` (`ctrl+enter` on Windows and Linux) to save and move to the next unannotated run, or click **Save**.


🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Document that a verdict is optional.

This step currently tells users to set a verdict, contradicting the shipped behavior described at Line 40: comments or tags can be saved independently. Reword the workflow to make the verdict optional and explicitly preserve the comment-only/tag-only path.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/reference/advanced/run-annotations/index.md` around lines 31 - 32, Update the annotation workflow step before the optional fields to state that selecting a verdict is optional. Explicitly mention that users can save comments or tags without selecting a verdict, while preserving the existing save and advance instructions.

coderabbitai Bot requested changes Jul 18, 2026

View reviewed changes

Comment thread docs/reference/advanced/run-annotations/index.md Outdated

coderabbitai Bot approved these changes Jul 18, 2026

View reviewed changes

claude added 2 commits July 18, 2026 03:11

Avoid minimizing "just" in annotations docs

47008c3

Style guide flags "just" as filler; use "only". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A8uqUP5UYj7vEmofWe4mHW

This was referenced Jul 21, 2026

docs: role on-ramps and a verified cookbook #2096

Closed

docs: role-based get-started, a cookbook of recipes, and an evaluate section #2102

Merged

strawgate enabled auto-merge (squash) July 23, 2026 23:26

Merge branch 'main' into claude/annotations-feature-flag-rollout-9g08iu

7ff7a93

coderabbitai Bot requested changes Jul 23, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add docs for annotating agent runs#2085

Add docs for annotating agent runs#2085
strawgate wants to merge 5 commits into
mainfrom
claude/annotations-feature-flag-rollout-9g08iu

strawgate commented Jul 18, 2026 •

edited by cubic-dev-ai Bot

Loading

Uh oh!

coderabbitai Bot commented Jul 18, 2026 •

edited

Loading

Reviews paused

Walkthrough

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Uh oh!

coderabbitai Bot Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		4. Add any of the optional fields below. Which ones appear depends on the verdict you picked.
		5. Save. Press `cmd+enter` (`ctrl+enter` on Windows and Linux) to save and move to the next unannotated run, or click Save.

Uh oh!

Conversation

strawgate commented Jul 18, 2026 • edited by cubic-dev-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this adds

Page contents

Uh oh!

coderabbitai Bot commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

strawgate commented Jul 18, 2026 •

edited by cubic-dev-ai Bot

Loading

coderabbitai Bot commented Jul 18, 2026 •

edited

Loading