ci: run property-based tests nightly with a large max_examples#4128
Merged
Conversation
Hypothesis settings profiles replace the hardcoded per-test max_examples: "default" (200) for PR CI and local runs, "nightly" (10,000) selected via the HYPOTHESIS_PROFILE environment variable. A new scheduled workflow runs tests/properties nightly at the large profile, plus a second pass under pytest-run-parallel (4 threads), which is how the bug behind scikit-hep#3962 was originally found. A README badge surfaces nightly failures; the stale Build Test badge (dead since the rename in scikit-hep#3969) now points at ci.yml. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
4 tasks
ianna
approved these changes
Jun 18, 2026
ianna
left a comment
Member
There was a problem hiding this comment.
@TaiSakuma - Thanks! It looks good to me. Please go ahead and merge it if you are done with it. Thanks.
ikrommyd
reviewed
Jun 18, 2026
ikrommyd
left a comment
Member
There was a problem hiding this comment.
I'm not particularly good with how conftest works in this case. So this does not increase the max_examples when we run the whole tests right? And only increases it in this particular property based testing CI?
ikrommyd
approved these changes
Jun 18, 2026
ikrommyd
left a comment
Member
There was a problem hiding this comment.
Yeah I'm pretty sure it does what my above comment says. Good
TaiSakuma
added a commit
to TaiSakuma/awkward
that referenced
this pull request
Jun 18, 2026
Check awkward_BitMaskedArray_to_ByteMaskedArray against its pure-Python reference implementation from kernel-specification.yml over Hypothesis-generated inputs, on the CPU kernel and (with `-m cuda`) the CUDA kernel. Example budget and deadline come from the central profile in scikit-hep#4128. This is a pilot on a single kernel; if it works well, the approach will be extended to the remaining kernels. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
TaiSakuma
added a commit
that referenced
this pull request
Jun 19, 2026
Check awkward_BitMaskedArray_to_ByteMaskedArray against its pure-Python reference implementation from kernel-specification.yml over Hypothesis-generated inputs, on the CPU kernel and (with `-m cuda`) the CUDA kernel. Example budget and deadline come from the central profile in #4128. This is a pilot on a single kernel; if it works well, the approach will be extended to the remaining kernels. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
As discussed in #3962 (comment): keep Hypothesis
max_examplessmall for PR CI, and run the property-based tests on a schedule with a largemax_examples. The bug fixed in #3962 was within reach of the existing property tests but was only stumbled on locally — PR-level example counts are too small for discovery, and raising them for every PR (and every matrix entry) would be too slow.What this does
tests/properties/conftest.pyreplace the hardcoded per-test@settings(max_examples=...):default:max_examples=200— used by PR CI and local runs (~4 s for the current suite, measured ~5.4 ms/example).nightly:max_examples=10_000,deadline=None,print_blob=True,derandomize=False— ~1 min/test locally, an estimated ~6–9 min on a CI runner.HYPOTHESIS_PROFILEenvironment variable. (The--hypothesis-profileCLI flag can't be used: profiles registered in a sub-conftest aren't visible atpytest_configuretime.)derandomize=Falseis explicit because on GitHub Actions Hypothesis auto-loads its built-inciprofile (derandomize=True), which the nightly profile would otherwise inherit — and then every nightly run would test the same 10,000 examples. PR CI stays derandomized, same as before.property-tests.yml(daily at 3:34 UTC +workflow_dispatch):awkward-cppwheel with the same steps and pinned action SHAs asreusable-test.yml, then runspytest tests/propertiesat thenightlyprofile.pytest-run-parallel --parallel-threads 4at thedefaultprofile — this is how the bug behind fix: proper comparison inak.almost_equalbetween record array fields #3962 was originally found, and the property tests currently never run threaded in CI (the 3.14t matrix entries userequirements-test-nogil.txt, which doesn't includehypothesis-awkward). It runs even if the serial pass fails, so one night reports both results.test.yml(HTTP 404) since the reusable-workflows split in ci: use reusable workflows #3969 — it now points atci.yml.Net effect on PR CI
test_to_from_buffers.py::test_roundtripdrops from 1000 to 200 examples in PR runs; thearray_equaltests stay at 200. Deep exploration moves to the nightly run, which uses fresh random seeds each night (~3.6 M examples/test/year cumulatively).Test plan
pytest tests/propertiespasses at the default profile (3.74 s vs 7.99 s before — confirms 200/test)HYPOTHESIS_PROFILE=nightlyruns 10,000 examples (56 s for one test)HYPOTHESIS_PROFILEvalue fails loudly (InvalidArgument)reusable-test.ymlgh workflow run property-tests.yml(the cron only fires from the default branch)Possible follow-ups (not in this PR)
database=Nonevia the inheritedciprofile;print_blob=Truealready gives@reproduce_failureblobs for local reproduction).