Feature request: optional verbosity during Suite.run (per check / per generation)

[davidberenstein1957]

Summary

When a Suite contains multiple scenarios and each scenario applies several checks (including LLM-based checks with multiple generations), await suite.run(target=...) can run for a long time with little or no incremental feedback in the terminal or logs.

Request

Add a way to surface progress and identity for each unit of work during execution, for example:

• Which scenario is currently running (name / id).
• Which check (or step) within that scenario is executing.
• For generator-backed checks: optional visibility into generation rounds or at least start/complete per check (without necessarily dumping full prompts/responses by default).

This could take the form of:

• A verbose= / progress= flag on Suite.run(), or
• Structured logging hooks / callbacks (e.g. on_scenario_start, on_check_start, on_generation_complete), or
• Integration with Python logging at INFO/DEBUG levels.

Motivation

• CI and scripts: Easier to see where a run is stuck or slow.
• Demos and notebooks: Clearer narrative when multiple checks are composed.
• Debugging: Faster isolation of failing checks without waiting for the full SuiteResult.

Notes

• Default behavior should remain quiet for backward compatibility; verbosity should be opt-in.
• If full prompt/response logging is supported, it should be a separate, explicit opt-in to avoid leaking sensitive data.

---

Happy to help refine API shape or contribute a PR if maintainers agree on the direction.

[linear]

OSS-67

[demoiverrakada]

Hi @davidberenstein1957, I would love to pick this up as my first contribution!
I plan to add a verbose=False flag to Suite.run() and use standard Python logging (at the INFO level) to surface the current scenario and check. I will make sure it stays quiet by default and doesn't dump sensitive prompt data.
Let me know if this sounds good or if you'd prefer the callback hooks approach instead, and I'll open a draft PR!

[davidberenstein1957]

Hi @demoiverrakada feel free to give it a try. Do not forget that the tests run async, so you might need to consider that :)

[demoiverrakada]

Hi @davidberenstein1957 — opened the PR: #2607. Went with the Python-logging option; scenarios run concurrently under parallel=True, so each log line includes the scenario name to keep interleaved output readable. Happy to address any feedback!