fix(evals): reject non-positive OnlineEvaluator.max_concurrency

[VectorPeak]

• Closes OnlineEvaluator should reject non-positive max_concurrency #6266

This rejects invalid non-positive OnlineEvaluator.max_concurrency values before they create a semaphore that cannot run evaluations.

What Problem This Solves

OnlineEvaluator.max_concurrency is a user-facing configuration value for bounding how many online evaluations may run at the same time. A positive value has a clear meaning: 1 allows one in-flight evaluation, 2 allows two, and so on. 0, however, does not provide a useful concurrency budget for this API.

Today, OnlineEvaluator.__post_init__ passes the value directly into threading.Semaphore(self.max_concurrency). That creates two different failure modes for the same class of invalid input:

• max_concurrency=-1 fails immediately with Python's low-level ValueError: semaphore initial value must be >= 0.

• max_concurrency=0 is accepted because threading.Semaphore(0) is valid, but it starts with zero available permits.

The second case is the confusing one. Once the evaluator is attached, the online evaluation dispatch path checks capacity with:

online_eval.semaphore.acquire(blocking=False)

For a zero-permit semaphore, that call always returns False. As a result, every sampled evaluation for that OnlineEvaluator is treated as if the evaluator is already at its concurrency limit. The run does not fail at configuration time; it simply follows the documented ?limit reached? path.

That makes the bug easy to miss in real usage. If an on_max_concurrency callback is configured, users may see every evaluation reported as dropped even though there is no real concurrency pressure. If no callback is configured, those evaluations are silently ignored, which can make online evaluation appear to be installed while producing no results.

This is different from intentionally disabling evaluation. The API already has an explicit sampling control for that: sample_rate=0.0. By contrast, max_concurrency should describe the positive number of evaluations that may run concurrently after an evaluation has been sampled.

This change also aligns OnlineEvaluator with the existing validation in Dataset.evaluate(..., max_concurrency<=0), which already rejects non-positive concurrency limits before constructing a semaphore.

Change

This change adds explicit validation for OnlineEvaluator.max_concurrency inside OnlineEvaluator.__post_init__, before the class creates its internal threading.Semaphore.

That placement is intentional. __post_init__ is the point where the dataclass turns user-supplied configuration into runtime state, so it is the narrowest place to reject invalid concurrency settings before they become background-dispatch behavior. A positive value still means the same thing as before: the number of online evaluations that may run at the same time. Values below 1 now fail fast because they do not represent a useful concurrency limit for this API.

The new guard is intentionally small:

if self.max_concurrency < 1:
    raise ValueError(f'max_concurrency must be >= 1, got {self.max_concurrency}')

This gives the two invalid boundaries the same user-facing treatment:

• max_concurrency=0 now raises immediately instead of creating a zero-permit semaphore. Without the guard, every later acquire(blocking=False) call fails and every sampled online evaluation looks like it hit a real concurrency limit.
• max_concurrency=-1 now raises the same OnlineEvaluator validation error instead of leaking Python's lower-level threading.Semaphore validation message.

Positive concurrency limits are unchanged. The default 10, the single-worker case 1, and higher limits still create the same semaphore-backed limiter. The existing on_max_concurrency path is still used when a valid positive limit is reached by real concurrent demand.

The test update covers both invalid inputs, 0 and -1, so the API now has one consistent rule: OnlineEvaluator.max_concurrency must be a positive integer, while intentional disabling remains represented by the existing sampling control such as sample_rate=0.0.

Evidence

Before this change, the underlying semaphore behavior is:

threading.Semaphore(0) constructed=True first_acquire=False

threading.Semaphore(-1) error=ValueError: semaphore initial value must be >= 0

threading.Semaphore(1) constructed=True first_acquire=True

That means max_concurrency=0 is accepted at construction time but can never acquire a permit in the online evaluation dispatch path.

After this change, OnlineEvaluator(..., max_concurrency=0) and OnlineEvaluator(..., max_concurrency=-1) fail eagerly with:

ValueError: max_concurrency must be >= 1, got <value>

Possible call chain / impact

OnlineEvaluator(max_concurrency=0)

  -> OnlineEvaluator.__post_init__()

  -> threading.Semaphore(0)

  -> online evaluation dispatch

  -> online_eval.semaphore.acquire(blocking=False)

  -> False every time

  -> on_max_concurrency callback or silent drop

This PR only changes invalid OnlineEvaluator configuration. It does not change sampling, sink emission, on_max_concurrency behavior for real concurrency pressure, or positive concurrency limits.

Validation

• uv run --project D:\ZXY\Github\pydantic-ai\pydantic_evals pytest D:\ZXY\Github\pydantic-ai\tests\evals\test_online.py -k "online_evaluator_invalid_max_concurrency or online_evaluator_custom_config or max_concurrency_respected" - passed, 4 tests

• git diff --check - passed

Checklist

• Any AI generated code has been reviewed line-by-line by the human PR author, who stands by it.

• No breaking changes in accordance with the version policy.

• PR title is fit for the release changelog.

[coderabbitai]

📝 Walkthrough

Walkthrough

OnlineEvaluator.__post_init__ now validates max_concurrency before constructing the semaphore, raising ValueError when the value is less than 1. Previously, values like 0 or negative numbers were passed directly to threading.Semaphore, causing silent evaluation drops or a less clear error. A new parametrized test verifies the validation error is raised with the expected message for max_concurrency values of 0 and -1.

Changes

Area	Change
pydantic_evals/pydantic_evals/online.py	Added early validation in __post_init__ rejecting max_concurrency < 1
tests/evals/test_online.py	Added parametrized test for invalid max_concurrency values

Possibly related PRs

• pydantic/pydantic-ai#6228: Introduces the same fail-fast max_concurrency < 1 validation with an identical error message, applied to Dataset.evaluate instead of OnlineEvaluator.

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The code and test changes satisfy #6266 by rejecting max_concurrency values below 1 at construction time.
Out of Scope Changes check	✅ Passed	The PR stays focused on validation and tests, with no obvious unrelated or out-of-scope changes.
Title check	✅ Passed	Title clearly states the main change: rejecting non-positive OnlineEvaluator.max_concurrency values.
Description check	✅ Passed	Description matches the template: it links the issue and includes the required checklist items.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[dsfaccini]

David's AICA here: 🔍 Local review suite (automated, pre-merge)

Wave	Status	Blocking	Required (do)	Warnings (defer/skip)
1 · code-correctness	✅ closed	0	0	0
2 · tests/cassettes/docs	✅ closed	0	0	0
3 · cross-cutting/siblings	✅ closed	0	0	0

Verdict: ✅ all 3 waves passed, 0 blocking, 0 required
Reviewed at 2026-07-05 · diff 385a927c

⚠️ 2 informational (skip)

Automated review — flagged by Claude, not yet confirmed by a maintainer.

1. Backward-compat: max_concurrency=0 silent-drop → raise is a permitted bug fix · online.py · OnlineEvaluator.__post_init__
Pre-fix, max_concurrency=0 built a zero-permit threading.Semaphore(0) that made every acquire(blocking=False) return False, silently dropping all sampled evaluations; 0 has no documented meaning on the field. Rejecting it now is a bug fix to an undocumented assumption — version-policy.md treats this as not a breaking change — so no deprecation/changelog gymnastics needed. Skip — documented for awareness.

2. Precedent verified — this completes the #6228 pattern · online.py · OnlineEvaluator.__post_init__
The guard mirrors the merged Dataset.evaluate validation (dataset.py, added by #6228): byte-identical < 1 check, ValueError type, message string must be >= 1, got {…}, and the parametrize([0, -1]) test shape. Both user-facing max_concurrency→semaphore paths in pydantic_evals/ are now guarded — no mirror gap remains. The Dataset.evaluate field docstring likewise omits the >= 1 bound, so leaving it out of OnlineEvaluator's docstring is the consistent choice. Skip.