Skip to content

feat(checks): add parallel scenario execution to Suite.run#2474

Merged
kevinmessiaen merged 8 commits into
Giskard-AI:mainfrom
harsh21234i:feat/suite-run-parallel
May 21, 2026
Merged

feat(checks): add parallel scenario execution to Suite.run#2474
kevinmessiaen merged 8 commits into
Giskard-AI:mainfrom
harsh21234i:feat/suite-run-parallel

Conversation

@harsh21234i

Copy link
Copy Markdown
Contributor

Closes #2471

Summary

  • add a parallel: bool = False flag to Suite.run()
  • run scenarios concurrently with asyncio.TaskGroup when parallel=True
  • preserve result order to match scenario order
  • include the parallel flag in checks_suite_run_started and checks_suite_run_finished telemetry properties

Behavior

  • default behavior is unchanged: suite.run() still runs serially
  • suite.run(parallel=True) runs scenarios concurrently
  • when return_exception=False, parallel execution fails fast and cancels sibling tasks if one scenario raises
  • when return_exception=True, scenario-level exception handling is preserved through scenario.run(...)

Testing

  • uv run -m pytest -q libs/giskard-checks/tests/core/test_suite.py
  • uv run ruff check libs/giskard-checks/src/giskard/checks/scenarios/suite.py libs/giskard-checks/tests/core/test_suite.py

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces parallel execution for scenario suites using asyncio.TaskGroup, allowing users to run scenarios concurrently while maintaining result order. The changes include updates to the Suite.run method, telemetry tracking for the new parallel flag, and a suite of tests verifying performance gains and error handling. Feedback from the review suggests centralizing the telemetry property logic, adopting list comprehensions for more idiomatic code, and implementing a concurrency limit using a semaphore to avoid resource exhaustion during large-scale parallel execution.

Comment thread libs/giskard-checks/src/giskard/checks/scenarios/suite.py Outdated
Comment thread libs/giskard-checks/src/giskard/checks/scenarios/suite.py Outdated
Comment thread libs/giskard-checks/src/giskard/checks/scenarios/suite.py Outdated

@kevinmessiaen kevinmessiaen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your PR, looking good, a few comments.

Formatting is broken, make sure to fix by running make format

Comment thread libs/giskard-checks/src/giskard/checks/scenarios/suite.py Outdated
@harsh21234i

Copy link
Copy Markdown
Contributor Author

Updated the PR.

I fixed the parallel=True exception behavior so the common single-failure case re-raises the original exception
instead of surfacing a raw ExceptionGroup, and I also formatted the touched files.

make format is not available in my current Windows environment, so I ran the formatter directly with uv run ruff format on the changed files instead, then verified with ruff check and the focused suite tests.

Checks run:

  • uv run ruff format libs/giskard-checks/src/giskard/checks/scenarios/suite.py libs/giskard-checks/tests/core/ test_suite.py
  • uv run ruff check libs/giskard-checks/src/giskard/checks/scenarios/suite.py libs/giskard-checks/tests/core/ test_suite.py
  • uv run -m pytest -q libs/giskard-checks/tests/core/test_suite.py

@harsh21234i
harsh21234i requested a review from kevinmessiaen May 20, 2026 15:18
@kevinmessiaen

Copy link
Copy Markdown
Member

Thanks for your contribution @harsh21234i

I've refactored a bit the run method to have two separate methods for sequential/parallel

@kevinmessiaen
kevinmessiaen merged commit 56b1c9d into Giskard-AI:main May 21, 2026
25 of 30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

Suite.run(): add parallel flag to run scenarios concurrently

2 participants