docs: add proposed server-side policy evaluation tutorial

[pbeckham]

What

Documentation-driven development for the server-side Kosli Evaluate project (kosli-dev/server#6267). Adds a tutorial describing the proposed V1 kosli evaluate policy flow, published so it can be shared internally and with design-partner customers for feedback before the feature is built.

This is a design preview — nothing in it is runnable today. The page is marked as such throughout.

Scope (this draft)

Core happy path for a governance platform engineer:

• Author a policy locally — Rego with name/purpose/input-context schema (allow mandatory).
• Ship a versioned policy into Kosli — CLI/API, identifier + version.
• Evaluate and record a decision — kosli evaluate policy runs server-side and records a decision against a control; synchronous.
• Trace and replicate — evaluation context pins trail moments, attestation_ids, policy version, params, timestamp; includes the T / T+1 / T+2 temporal-correctness case.
• Evaluate a local policy file — decision still recorded server-side.

An explicit out-of-scope-for-V1 section keeps reviewer feedback on target (managing policies, testing/validation, reporting/audit views, unhappy-path inspection, composable policies, non-trail targets, scheduled evals, risk measurement).

Design notes surfaced for feedback

• Command surface verified against the real kosli attest decision and kosli evaluate trail flags. --compliant and evidence (--attachments/--user-data) are intentionally not flags — the server derives compliance and captures the policy/report/violations as output.
• The publish command reuses kosli create policy, which collides with the existing environment-policy command — flagged inline as an open question.
• Rego input shape is borrowed from the client-side tutorial and may differ server-side.

Draft because

Content describes proposed behavior and will change as the design firms up; opening as draft to gather review before it's merged/published.

🤖 Generated with Claude Code

[claude]

Claude finished @pbeckham's task in 3m 3s —— View job

---

Review: Server-side policy evaluation tutorial

• Read CLAUDE.md and doc-review skill
• Gather PR diff and read changed files
• Check navigation config for new pages
• Review content quality, structure, and conventions
• Post inline comments on specific issues
• Post final summary

---

Summary

Strong design-preview tutorial that clearly separates what's proposed from what exists today. The <Warning> banner, the "Why server-side" comparison table, and the explicit "out of scope" section are well done — they set the right expectations for a design preview. The working_with_controls.mdx updates (CLI commands, coverage tab, archiving, flow template section) are clean additions.

Findings

By file: tutorials/evaluate_policies_server_side.mdx

Category	Issue
Improvement	Meta description ~170 chars — trim to ≤160 for search engines.
Improvement	Intro/structure mismatch — five numbered parts listed, but only part 1 uses <Steps>. Parts 2–5 read as reference sections, not tutorial steps. Reframe the intro to set that expectation.
Improvement	kosli create policy collision not called out explicitly — the <Info> callout hints at the relationship but doesn't name the naming clash. For a design preview seeking feedback, make it unmissable.
Improvement	Async/--sync/--assert restated — paragraph at lines 204–206 repeats what the intro and flags table already say. Remove it.
Suggestion	--trail flag undocumented by example — appears in the flags table but not in any command example. A brief usage note would help.
Improvement	Placeholder images invisible on preview — all three policy-*.png files are 70-byte transparent PNGs. They render as empty frames on the Mintlify preview. Use visible placeholders or remove the <Frame> blocks until real screenshots are ready.

By file: tutorials/working_with_controls.mdx

No new issues. The additions (CLI examples, archiving section, coverage tab, flow template section) are well-structured and consistent with the existing page.

By file: config/navigation.json

No issues. tutorials/evaluate_policies_server_side is correctly listed under the "Evaluation" subgroup.

CLAUDE.md compliance

Rule	Status
Root-relative links only	All links are root-relative
New page added to config/navigation.json	Yes — under Tutorials > Evaluation
YAML frontmatter present	Yes — title and description
Writing style (active voice, imperative mood)	Good throughout
American spelling	Consistent

Diátaxis assessment

The page is titled as a tutorial but is a hybrid: part 1 (the <Steps> section) is a genuine tutorial walk-through, while parts 2–5 read more like reference/how-to material. This is fine for a design preview — but if the page ships to production, consider splitting or reframing to make the boundary explicit.

Overall

6 findings (0 critical, 4 improvement, 2 suggestion). The content is well-written, the design-preview framing is clear, and the compliance checks pass. The main action items are tightening the structure/intro mismatch and making the kosli create policy naming collision explicit for reviewers.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
kosli	🟢 Ready	View Preview	Jul 23, 2026, 1:43 PM

[claude]

Improvement: The intro says evaluation is "asynchronous by default" (line 144) and this is repeated almost verbatim in the paragraph at lines 181-183. Consider removing one to avoid the redundancy — the flag table already documents both --sync and --assert.

[claude]

Improvement: All three policy-*.png images are identical 1×1 pixel transparent PNGs. The {/* TODO */} comments make clear these are placeholders, which is fine for a design preview — but readers on the Mintlify preview site will see empty frames with no visual indication that they're placeholder slots. Consider either:

• Using a visible placeholder image (e.g. a solid grey rectangle with "Mockup coming soon" text), or
• Removing the <Frame> blocks entirely until real screenshots are ready, and noting "Screenshots will be added when the UI is built" in the text.

The current approach renders as invisible content on the live preview.

[claude]

Improvement: This paragraph (lines 202–205) repeats the async-by-default / --sync / --assert information from the intro paragraph at lines 165–167, and the flags table at lines 199–200 already documents both flags with descriptions. Consider removing this paragraph — the table plus the first mention are sufficient.

Suggested change

	By default the command queues the evaluation and returns without waiting; the decision is
	recorded when the evaluation completes. Add `--sync` to wait for the outcome, or `--assert` to
	both wait and fail the step when the policy denies — use `--assert` as a pipeline gate. Once
	recorded, a decision appears against the policy — see
	[View the decisions for a policy](#view-the-decisions-for-a-policy).
	recorded, a decision appears against the policy — see

[claude]

Improvement: The callout mentions the naming question vaguely ("how it relates to existing environment-policy commands") but doesn't name the collision. kosli create policy already exists in the CLI (it creates environment policies — see client_reference/kosli_create_policy). Since this is a design preview seeking feedback, calling out the collision explicitly would help reviewers spot it:

Suggested change

	In the beta, authoring is **CLI/API only** — the Kosli UI is read/tag-only. The exact publish
	command surface (and how it relates to existing environment-policy commands) is part of what
	we want feedback on.
	In the beta, authoring is **CLI/API only** — the Kosli UI is read/tag-only. The publish
	command shown here reuses `kosli create policy`, which **today** creates environment
	policies — whether this should be a new command or the existing one should be overloaded
	is an open design question we want feedback on.

[claude]

Suggestion: The intro lists five numbered steps as the tutorial flow, but only steps within "Author, ship, and evaluate a policy" use the <Steps> component. The remaining four (Validate, Manage, View decisions, Inspect failures) are standalone ## sections. This creates a mismatch between the promised structure and the page layout.

Consider either:

• Reframing the intro: "This tutorial walks through the core loop (step 1), then covers reference topics for…"
• Or restructuring so the <Steps> sections and standalone sections are introduced distinctly.

[claude]

Suggestion: --trail is documented here but never appears in any of the example commands on this page. A brief example or note showing when it's needed (e.g. "when evaluating multiple trails, add --trail release-456 to specify where the decision is recorded") would help readers understand the flag's purpose.

[claude]

Suggestion: --evaluation-context carries different semantics in kosli create policy (line 146: a schema type — trail or trails) vs kosli evaluate policy (here: the actual trail name). A reader who just saw --evaluation-context trail in the create command may trip over the overload. Consider a short clarification in the flag table, e.g. "The trail name (or names) whose facts are evaluated — not the schema declaration from kosli create policy."

(This may also be useful design feedback for the CLI surface itself.)

[claude]

Improvement: Lines 204–206 restate the async-by-default / --sync / --assert information from lines 166–168. The flags table at lines 200–201 already documents both flags with descriptions. Consider removing this paragraph — the table plus the first mention are sufficient.

(Flagged in prior reviews as well but still present.)

[claude]

Improvement: All three policy-*.png images are 70-byte, 1×1 transparent PNGs. On the Mintlify preview they render as invisible <Frame> blocks. Consider either using a visible placeholder (grey rectangle with "Mockup coming soon") or removing the <Frame> blocks entirely until real screenshots are ready, noting "Screenshots will be added when the UI is built" in the text.

[claude]

Improvement: This paragraph (lines 204–206) restates the async-by-default / --sync / --assert behavior already covered in the intro (lines 166–168) and the flags table (lines 200–201). Consider removing lines 204–207 — the table and the first mention are sufficient.

Suggested change

	By default the command queues the evaluation and returns without waiting; the decision is
	recorded when the evaluation completes. Add `--sync` to wait for the outcome, or `--assert` to
	both wait and fail the step when the policy denies — use `--assert` as a pipeline gate. Once
	recorded, a decision appears against the policy — see
	[View the decisions for a policy](#view-the-decisions-for-a-policy).
	recorded, a decision appears against the policy — see

[claude]

Suggestion: The meta description is ~170 characters — search engines truncate beyond 160. Consider trimming:

Suggested change

	title: "Evaluate policies server-side"
	description: "Author and ship a versioned Rego policy into Kosli, then evaluate it server-side to record a trustworthy, replicable decision."

[claude]

Improvement: The intro lists five numbered parts, but only part 1 ("Author, ship, and evaluate") uses the <Steps> component — parts 2–5 are standalone ## sections. This creates a structural mismatch between the promised flow and the page layout.

Consider reframing the intro to distinguish the core tutorial (part 1) from the supporting reference sections (parts 2–5), e.g.: "This tutorial walks through the core loop (part 1). The sections that follow cover supporting operations."

[claude]

Improvement: kosli create policy already exists in the CLI — it creates environment policies (see client_reference/kosli_create_policy). The callout hints at the relationship but doesn't name the collision explicitly. Since this is a design preview seeking feedback, making the clash unmissable would be more helpful:

Suggested change

	`kosli create policy` is the same command used for [environment policies](/getting_started/policies):
	`--type rego` selects an evaluation policy, while the default `--type env` creates an environment
	policy. In the beta, authoring is **CLI/API only** — the Kosli UI is read/tag-only.
	`kosli create policy` is the same command used for [environment policies](/getting_started/policies):
	`--type rego` selects an evaluation policy, while the default `--type env` creates an environment
	policy. Whether reusing the same command is the right design — or whether evaluation policies
	should have their own command — is an **open design question** we want feedback on.
	In the beta, authoring is **CLI/API only** — the Kosli UI is read/tag-only.

[claude]

Suggestion: --trail is documented in the flags table here but never appears in any example command on the page. A brief example or note showing when it's needed — e.g. "when evaluating a trail in another flow and recording the decision on a different trail, add --trail release-456" — would make the flag's purpose concrete.

[AlexKantor87]

Not sure i understand the ('pr approved' here) part?

[AlexKantor87]

Ah i understand it's for the example further below.

[AlexKantor87]

There's a few of these in the doc that confused me that we could change to "in the example below" but it's non-material to this for now

[AlexKantor87]

As written I think this example gets rejected by the parameter validation described later on. The Parameters section says a missing parameter is rejected before the evaluation is queued?

[AlexKantor87]

Further down, "Inspect a failing evaluation" says a missing attestation surfaces as a violation, but with this policy it doesn't: a missing slot leaves this rule undefined too, so you get a denial with an empty violations list and nothing to fix. A second rule for the missing slot keeps that claim true:

Suggested change

	allow if {
	every pr in input.trail.compliance_status.attestations_statuses[pr_attestation_name].pull_requests {
	count(pr.approvers) > 0
	}
	}
	violations contains msg if {
	some pr in input.trail.compliance_status.attestations_statuses[pr_attestation_name].pull_requests
	count(pr.approvers) == 0
	msg := sprintf("pull-request %v has no approvers", [pr.url])
	}
	allow if {
	every pr in input.trail.compliance_status.attestations_statuses[pr_attestation_name].pull_requests {
	count(pr.approvers) > 0
	}
	}
	violations contains msg if {
	some pr in input.trail.compliance_status.attestations_statuses[pr_attestation_name].pull_requests
	count(pr.approvers) == 0
	msg := sprintf("pull-request %v has no approvers", [pr.url])
	}
	violations contains msg if {
	not input.trail.compliance_status.attestations_statuses[pr_attestation_name]
	msg := sprintf("expected attestation %v is missing from the trail", [pr_attestation_name])
	}

[AlexKantor87]

There's a few of these in the doc that confused me that we could change to "in the example below" but it's non-material to this for now

[AlexKantor87]

seemingly opa fmt would re-write this. Consistency nit only

Suggested change

	default allow = false
	default allow := false

[AlexKantor87]

So how would we evaluate a SARIF file - is that still to be done client side and then our loaded policies only check for if it ran and how many issues it had?

[AlexKantor87]

This is directly below the first example block of text and talks about trails but it's not obvious as to why trails suddenly were mentioned as flows in there and artifact fingerprint.

Should we consider saying how the trails are chosen / found?

[jumboduck]

This link was not working