chore: generate kernel artefacts from CMake instead of nox -s prepare

[henryiii]

🤖 AI text below 🤖

Building awkward-cpp from the repository no longer needs nox -s prepare — a fresh clone builds with just pip install ./awkward-cpp.

• CMake runs the generation scripts at configure time (copy-cpp-headers.py, generate-kernel-signatures.py) when the repository sources are present, guarded by a content hash of the inputs (registered as CMAKE_CONFIGURE_DEPENDS, so editing the kernel spec retriggers via a plain rebuild). An sdist ships the artefacts and skips generation; -DAWKWARD_PREPARE=OFF opts out.
• Kernel tests are generated only when packing the sdist (SKBUILD_STATE=sdist): they aren't needed to build a wheel, and generating them needs numpy, which pyodide's cross-build environment can't provide host-side. Developers get them from nox -s prepare -- --tests as before.
• sdist.cmake = true runs configure before packing, so sdists built from a raw checkout still contain the generated files. pyyaml is a build requirement when not building from an sdist (if.from-sdist = false override), with numpy added for the sdist state.
• Generation is rebuild-friendly: scripts leave outputs untouched when content is unchanged (ignoring the AUTO GENERATED ON stamp), and header copies preserve source mtimes, so regeneration does not invalidate incremental builds.
• Anchored sdist.include patterns: the bare "header-only" pattern also matched build/*/header-only and leaked configure outputs into the sdist once a build directory existed.
• Fixed a pre-existing yaml loader bug in dev/generate-tests.py: on Python < 3.13 it unconditionally used yaml.CSafeLoader, which doesn't exist when pyyaml lacks libyaml (seen on windows-11-arm).
• CI: dropped the prepare step from the awkward-cpp sdist/cibuildwheel jobs. It remains where the build cannot cover it: the pure-Python awkward package build (hatchling), docs data (--docs), and test workflows that need the generated kernel tests or cache the awkward-cpp wheel.

Verified locally beyond CI: clean-checkout sdist build (contents complete, tests included, no build/ leakage), wheel built from a fresh-clone state without test generation, wheel built from the sdist with generation skipped, and direct-CMake incremental behavior (reconfigure is a no-op; content changes to the spec regenerate; touch does not).

Note for review: awkward-cpp/pyproject.toml changed, so the "Needs C++ Release" check will flag a version bump for the next release.

[henryiii]

🤖 AI text below 🤖

What changed

awkward-cpp/CMakeLists.txt — the old "run nox -s prepare first" fatal error is replaced by generation logic: when the repository sources are present (kernel-specification.yml, dev/ scripts, header-only/), configure runs copy-cpp-headers.py, generate-kernel-signatures.py, and generate-tests.py. It's guarded by a content hash of all inputs stored in the CMake cache, and the inputs are registered as CMAKE_CONFIGURE_DEPENDS, so editing the kernel spec and running ninja reconfigures and regenerates automatically. An sdist (which ships the artifacts but not the sources) skips generation, and -DAWKWARD_PREPARE=OFF opts out entirely.

awkward-cpp/pyproject.toml — sdist.cmake = true makes scikit-build-core run configure before packing the sdist, so sdists built from a raw checkout still ship the generated files. A [[tool.scikit-build.overrides]] with if.from-sdist = false adds pyyaml/numpy as build requirements only when building from the repo — sdist consumers don't pay for them. I also anchored the sdist.include patterns (/header-only, …): the unanchored "header-only" pattern was pulling files from build/cpython-*/header-only/ into the sdist once a build directory existed.

dev/copy-cpp-headers.py and dev/generate-kernel-signatures.py — now leave outputs untouched when the content is unchanged (comparison ignores the "AUTO GENERATED ON" stamp line; header copies use copy2 to preserve source mtimes). This matters because generation now runs on every fresh configure: without it, kernels.h and the copied headers would get new mtimes each build and force a full kernel recompile. Verified: reconfigure + ninja is a no-op; a touch of the spec doesn't regenerate (content-based); a real content change does.

CI — removed the prepare step from the jobs that build/package awkward-cpp: the sdist and cibuildwheel jobs in reusable-build-wheels.yml and reusable-packaging-test.yml, plus the sdist-check jobs in deploy.yml and needs-cpp-release.yml. The kernel-test suites are also generated by the build, and cibuildwheel forwards SOURCE_DATE_EPOCH into containers, so generated-file stamps stay reproducible.

Docs — README, CONTRIBUTING.md, and AGENTS.md updated (dev setup is now just pip install ./awkward-cpp + pip install -e .; the no-isolation instructions gain pyyaml numpy).

What still uses nox -s prepare

• The pure-Python awkward package build (deploy/packaging-test "build awkward" jobs): it's hatchling, not CMake, and its wheel/sdist ship src/awkward/_connect/header-only and the CUDA signature table. Covering that would need a hatchling build hook — happy to add one as a follow-up if you want prepare gone from packaging entirely.
• Docs data (--docs target) in the docs workflow.
• Test workflows (reusable-test.yml etc.): they cache the awkward-cpp wheel, so on a cache hit no build runs and the generated test dirs must come from somewhere.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.03%. Comparing base (f3255a3) to head (8435904).
⚠️ Report is 33 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

see 88 files with indirect coverage changes

[henryiii]

Ran /simplify. Putting the output below, since we might be able to do some of the skipped stuff later with some redesign.

🤖 AI text below 🤖

Fixed

1. Removed the CMake SHA256 stamp machinery (awkward-cpp/CMakeLists.txt, −29 lines net). Three of the four agents independently flagged this: the input-hashing, _AWKWARD_PREPARE_STAMP cache variable, and output-existence checks were a second change-detection layer on top of the idempotence this same PR built into the scripts (copy2 in copy-cpp-headers.py, write_if_changed in generate-kernel-signatures.py). The scripts now run unconditionally at configure; CMAKE_CONFIGURE_DEPENDS registration stays. I verified by configuring twice: the scripts total well under a second, and artefact mtimes are byte-for-byte stable across reconfigures, so no rebuild churn. This also removes the drift the reuse agent spotted (prepare_outputs listed only one of four test dirs) and self-heals hand-edited generated files, which the stamp never would.
2. AGENTS.md build-deps note now says pyyaml and numpy, matching CONTRIBUTING.md. I went the opposite way from the agent's suggestion (dropping numpy from CONTRIBUTING): the command uses --check-build-dependencies, and under the pinned scikit-build-core (<1.0) wheel builds dynamically request numpy — the pyproject comment says so itself — so the check fails without it.

Skipped (with reasons)

• Building awkward-cpp writes into src/awkward/ (CUDA signatures, header-only copies): intentional — the new fresh-clone dev flow (pip install ./awkward-cpp; pip install -e .) depends on those files being generated. A --cpp-only flag would break that.
• Shared write_if_changed/reproducible_datetime/yaml-loader helper across dev scripts, and write_if_changed in generate-tests.py: real inconsistency, but the duplication is pre-existing and generate-tests.py only runs at one-shot sdist packs, where mtime churn doesn't matter (especially now that the stamp it papered over is gone).
• The two pyproject overrides: the simplification agent checked the suspected redundancy itself and concluded they're both needed — agreed, left as is.
• CUDA test suites generated needlessly during awkward-cpp sdist packs (~17 MB, ~1.5 s): a suite-selection flag for generate-tests.py isn't worth the plumbing at that cost.
• Four repeated nox -s prepare -- --tests workflow steps: each replaces a previously identical step one-for-one and is still needed (cibuildwheel's test-command reads {package}/tests-* from the checkout); a composite action for a one-liner is over-engineering.
• Full nox -s prepare left in the pure-python awkward build jobs: two agents disagreed on how far it could be narrowed (the awkward sdist may ship tests-cuda-kernels), and it's pre-existing breadth outside this diff.

[github-actions]

The documentation preview is ready to be viewed at http://preview.awkward-array.org.s3-website.us-east-1.amazonaws.com/PR4190

[henryiii]

#4236 would make this a little simpler.