feat(checks): add parallel scenario execution to Suite.run#2474
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces parallel execution for scenario suites using asyncio.TaskGroup, allowing users to run scenarios concurrently while maintaining result order. The changes include updates to the Suite.run method, telemetry tracking for the new parallel flag, and a suite of tests verifying performance gains and error handling. Feedback from the review suggests centralizing the telemetry property logic, adopting list comprehensions for more idiomatic code, and implementing a concurrency limit using a semaphore to avoid resource exhaustion during large-scale parallel execution.
kevinmessiaen
left a comment
There was a problem hiding this comment.
Thanks for your PR, looking good, a few comments.
Formatting is broken, make sure to fix by running make format
|
Updated the PR. I fixed the
Checks run:
|
|
Thanks for your contribution @harsh21234i I've refactored a bit the run method to have two separate methods for sequential/parallel |
…edicated run methods
Closes #2471
Summary
parallel: bool = Falseflag toSuite.run()asyncio.TaskGroupwhenparallel=Trueparallelflag inchecks_suite_run_startedandchecks_suite_run_finishedtelemetry propertiesBehavior
suite.run()still runs seriallysuite.run(parallel=True)runs scenarios concurrentlyreturn_exception=False, parallel execution fails fast and cancels sibling tasks if one scenario raisesreturn_exception=True, scenario-level exception handling is preserved throughscenario.run(...)Testing
uv run -m pytest -q libs/giskard-checks/tests/core/test_suite.pyuv run ruff check libs/giskard-checks/src/giskard/checks/scenarios/suite.py libs/giskard-checks/tests/core/test_suite.py