refactor: migrate from_feather to pyarrow.ipc API

[pranjan2023]

This PR updates both ak_to_feather and ak_from_feather to use the non-deprecated pyarrow.ipc API, replacing the deprecated Feather APIs and resolving the associated deprecation warnings.

Changes

• ak.to_feather

  • Replaces the deprecated Feather writer with pyarrow.ipc.RecordBatchFileWriter.
  • Preserves the existing behavior while avoiding the deprecated API.

• ak.from_feather

  • Replaces the deprecated pyarrow.feather.FeatherReader with pyarrow.ipc.open_file and IpcReadOptions.
  • Applies 'Table.select(columns)' after reading to preserve the existing column selection semantics, including requested column order and duplicate column selections.

Together, these changes migrate Awkward's Feather I/O implementation to the supported IPC API while maintaining compatibility with the previous behavior.

Fixes #4199.

[codecov]

Codecov Report

❌ Patch coverage is 60.00000% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.93%. Comparing base (c90cb15) to head (926f7f5).
⚠️ Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
src/awkward/operations/ak_to_feather.py	52.17%	11 Missing ⚠️
src/awkward/operations/ak_from_feather.py	85.71%	1 Missing ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
src/awkward/operations/ak_from_feather.py	93.75% <85.71%> (-6.25%)	⬇️
src/awkward/operations/ak_to_feather.py	68.29% <52.17%> (-16.90%)	⬇️

[ianna]

Hi @pranjan2023 . Thanks for your contribution!

I noticed that the new implementation changes the read behavior as well as moving to the non-deprecated IPC API.

Previously:

pyarrow.feather.read_table(path, columns=columns, ...)

could apply column selection during the read. The new implementation always calls:

arrow_table = reader.read_all()

and then selects the requested columns afterward.

Since reverting is not an option, could you please test whether this change reproduces the coverage hang locally and whether thread usage is involved?

coverage run -m pytest -vv -s \
  tests/test_1620_layout_builders.py::test_data_frame_listoffset_integers

Then try the same test with Arrow threading disabled:

ARROW_NUM_THREADS=1 \
coverage run -m pytest -vv -s \
  tests/test_1620_layout_builders.py::test_data_frame_listoffset_integers

It would also be useful to run the Feather-related tests directly:

coverage run -m pytest -vv -s \
  tests/test_*.py -k "feather or from_feather"

If the single test passes, could you also try the full coverage job with durations enabled?

ARROW_NUM_THREADS=1 \
coverage run -m pytest -vv --durations=30

The CI process exits with code 129 (SIGHUP), so I am trying to determine whether the new read_all() path or PyArrow thread usage is causing the job to hang or become significantly slower.

[pranjan2023]

Hi, thanks for the detailed repro steps. I ran the requested tests locally and here's what I found.

Setup

Since test_1620_layout_builders.py::test_data_frame_listoffset_integers requires PyROOT (which isn't part of the standard Awkward development environment), I created a separate conda environment with ROOT installed from conda-forge. I matched the key package versions with my main development environment (pyarrow 25.0.0, pytest 9.1.1, numpy 2.4.6, coverage 7.15.0, etc.) so the results would be comparable.

Results

1. Single ROOT test

   coverage run -m pytest -vv -s \
     tests/test_1620_layout_builders.py::test_data_frame_listoffset_integers

   Result: Passed in 2.34 s. No hang.

2. Same test with Arrow threading disabled

   ARROW_NUM_THREADS=1 \
   coverage run -m pytest -vv -s \
     tests/test_1620_layout_builders.py::test_data_frame_listoffset_integers

   Result: Passed in 3.69 s. No hang.

3. Feather-related tests

   coverage run -m pytest -vv -s \
     tests/test_*.py -k "feather or from_feather"

   Result: 2 passed in approximately 9 s. No hang.

4. Full test suite under coverage

   ARROW_NUM_THREADS=1 \
   coverage run -m pytest -vv --durations=30 tests

   Note: Running this command without the trailing tests argument picks up the testpaths configured in pyproject.toml, which includes tests-cuda, tests-cuda-kernels, and tests-cuda-kernels-explicit. Those fail during collection with:

   ModuleNotFoundError: No module named 'cupy'
   

   Since cupy requires an NVIDIA CUDA environment that I don't have, I scoped the run to tests, which avoids those GPU-specific test directories.

   Result: 6386 passed, 38 skipped, 0 failed in 92.96 s. No hang and no errors.

   All tests that were previously skipped because ROOT was unavailable ran successfully. The remaining skipped tests are pre-existing environment-dependent skips (TensorFlow and PyTorch do not currently provide Python 3.14 wheels, cupy requires CUDA, plus a few existing version/platform-specific skips unrelated to this change).

Summary

I wasn't able to reproduce the CI hang or observe any meaningful slowdown under any of the combinations I tried:

• Single test vs. full test suite
• Default Arrow threading vs. ARROW_NUM_THREADS=1
• With and without ROOT installed

Based on these results, the read_all() followed by .select(columns) path does not appear to be the source of the CI SIGHUP (exit code 129), at least on my local setup.

I've attached the complete output from all four runs in awkward_feather_investigation.log.

Environment

• macOS (Darwin)
• Python 3.14.6 (ROOT test environment; main development environment is Python 3.14.2)
• PyArrow 25.0.0

Since I can't reproduce the issue locally, it may be worth comparing the CI environment (PyArrow version, operating system, resource limits, or timeout configuration) against my setup. If there's additional CI-side information available, such as the exact job configuration or a stack trace/thread dump from the hung process, I'd be happy to investigate further.

[pranjan2023]

I don't yet know the root cause. I ran the full local test suite (including the specific test_1620_layout_builders.py test, with and without ARROW_NUM_THREADS=1, and with PyROOT installed) multiple times and couldn't reproduce a hang, crash, or segfault locally . So whatever's happening on the GPU runner may be CI/environment-specific (GPU driver, CUDA memory limits, runner resource constraints) rather than a bug in the changed code itself, but I can't confirm that without more visibility into the actual GPU job's logs.

[ianna]

I don't yet know the root cause. I ran the full local test suite (including the specific test_1620_layout_builders.py test, with and without ARROW_NUM_THREADS=1, and with PyROOT installed) multiple times and couldn't reproduce a hang, crash, or segfault locally . So whatever's happening on the GPU runner may be CI/environment-specific (GPU driver, CUDA memory limits, runner resource constraints) rather than a bug in the changed code itself, but I can't confirm that without more visibility into the actual GPU job's logs.

Thanks for looking into it. It would be useful to understand why it is happening. It is possible that the job just times out. The question is by how much we expect to delay the ending and if it is reasonable. Thanks!

[github-actions]

The documentation preview is ready to be viewed at http://preview.awkward-array.org.s3-website.us-east-1.amazonaws.com/PR4204