Skip to content

ci: run property-based tests nightly with a large max_examples#4128

Merged
TaiSakuma merged 1 commit into
scikit-hep:mainfrom
TaiSakuma:property-tests-nightly
Jun 18, 2026
Merged

ci: run property-based tests nightly with a large max_examples#4128
TaiSakuma merged 1 commit into
scikit-hep:mainfrom
TaiSakuma:property-tests-nightly

Conversation

@TaiSakuma

Copy link
Copy Markdown
Member

As discussed in #3962 (comment): keep Hypothesis max_examples small for PR CI, and run the property-based tests on a schedule with a large max_examples. The bug fixed in #3962 was within reach of the existing property tests but was only stumbled on locally — PR-level example counts are too small for discovery, and raising them for every PR (and every matrix entry) would be too slow.

What this does

  • Hypothesis settings profiles in tests/properties/conftest.py replace the hardcoded per-test @settings(max_examples=...):
    • default: max_examples=200 — used by PR CI and local runs (~4 s for the current suite, measured ~5.4 ms/example).
    • nightly: max_examples=10_000, deadline=None, print_blob=True, derandomize=False — ~1 min/test locally, an estimated ~6–9 min on a CI runner.
    • Selected via the HYPOTHESIS_PROFILE environment variable. (The --hypothesis-profile CLI flag can't be used: profiles registered in a sub-conftest aren't visible at pytest_configure time.)
    • derandomize=False is explicit because on GitHub Actions Hypothesis auto-loads its built-in ci profile (derandomize=True), which the nightly profile would otherwise inherit — and then every nightly run would test the same 10,000 examples. PR CI stays derandomized, same as before.
  • New scheduled workflow property-tests.yml (daily at 3:34 UTC + workflow_dispatch):
    • Builds/caches the awkward-cpp wheel with the same steps and pinned action SHAs as reusable-test.yml, then runs pytest tests/properties at the nightly profile.
    • A second pass runs under pytest-run-parallel --parallel-threads 4 at the default profile — this is how the bug behind fix: proper comparison in ak.almost_equal between record array fields #3962 was originally found, and the property tests currently never run threaded in CI (the 3.14t matrix entries use requirements-test-nogil.txt, which doesn't include hypothesis-awkward). It runs even if the serial pass fails, so one night reports both results.
  • README badge for the new workflow, so scheduled failures are visible. While editing that line: the existing "Build Test" badge has pointed at the deleted test.yml (HTTP 404) since the reusable-workflows split in ci: use reusable workflows #3969 — it now points at ci.yml.

Net effect on PR CI

test_to_from_buffers.py::test_roundtrip drops from 1000 to 200 examples in PR runs; the array_equal tests stay at 200. Deep exploration moves to the nightly run, which uses fresh random seeds each night (~3.6 M examples/test/year cumulatively).

Test plan

  • pytest tests/properties passes at the default profile (3.74 s vs 7.99 s before — confirms 200/test)
  • HYPOTHESIS_PROFILE=nightly runs 10,000 examples (56 s for one test)
  • An unknown HYPOTHESIS_PROFILE value fails loudly (InvalidArgument)
  • pre-commit green (incl. workflow schema validation and zizmor); actionlint reports only the same info-level SC2086 already present in reusable-test.yml
  • After merge: smoke-test with gh workflow run property-tests.yml (the cron only fires from the default branch)

Possible follow-ups (not in this PR)

  • Persist a Hypothesis example database across nightly runs so past failures are replayed first (currently database=None via the inherited ci profile; print_blob=True already gives @reproduce_failure blobs for local reproduction).
  • Run the threaded pass on free-threaded 3.14t for more aggressive concurrency coverage.

Hypothesis settings profiles replace the hardcoded per-test
max_examples: "default" (200) for PR CI and local runs, "nightly"
(10,000) selected via the HYPOTHESIS_PROFILE environment variable.

A new scheduled workflow runs tests/properties nightly at the large
profile, plus a second pass under pytest-run-parallel (4 threads),
which is how the bug behind scikit-hep#3962 was originally found. A README
badge surfaces nightly failures; the stale Build Test badge (dead
since the rename in scikit-hep#3969) now points at ci.yml.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions github-actions Bot added the type/ci PR title type: ci (set automatically) label Jun 12, 2026
@TaiSakuma
TaiSakuma requested review from ianna and ikrommyd June 12, 2026 21:00
@codecov

codecov Bot commented Jun 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.31%. Comparing base (b025b0b) to head (f8e1310).

Additional details and impacted files

see 1 file with indirect coverage changes

@ianna ianna left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TaiSakuma - Thanks! It looks good to me. Please go ahead and merge it if you are done with it. Thanks.

@ikrommyd ikrommyd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not particularly good with how conftest works in this case. So this does not increase the max_examples when we run the whole tests right? And only increases it in this particular property based testing CI?

@ikrommyd ikrommyd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I'm pretty sure it does what my above comment says. Good

@TaiSakuma
TaiSakuma merged commit 95fcb30 into scikit-hep:main Jun 18, 2026
39 checks passed
TaiSakuma added a commit to TaiSakuma/awkward that referenced this pull request Jun 18, 2026
Check awkward_BitMaskedArray_to_ByteMaskedArray against its pure-Python
reference implementation from kernel-specification.yml over
Hypothesis-generated inputs, on the CPU kernel and (with `-m cuda`) the
CUDA kernel. Example budget and deadline come from the central profile
in scikit-hep#4128.

This is a pilot on a single kernel; if it works well, the approach will
be extended to the remaining kernels.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
TaiSakuma added a commit that referenced this pull request Jun 19, 2026
Check awkward_BitMaskedArray_to_ByteMaskedArray against its pure-Python
reference implementation from kernel-specification.yml over
Hypothesis-generated inputs, on the CPU kernel and (with `-m cuda`) the
CUDA kernel. Example budget and deadline come from the central profile
in #4128.

This is a pilot on a single kernel; if it works well, the approach will
be extended to the remaining kernels.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@TaiSakuma
TaiSakuma deleted the property-tests-nightly branch July 21, 2026 21:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type/ci PR title type: ci (set automatically)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants