Skip to content

perf: optimize option-type ListArray to Arrow conversion#4207

Open
aashirvad08 wants to merge 2 commits into
scikit-hep:mainfrom
aashirvad08:arrow-string-null-alloc
Open

perf: optimize option-type ListArray to Arrow conversion#4207
aashirvad08 wants to merge 2 commits into
scikit-hep:mainfrom
aashirvad08:arrow-string-null-alloc

Conversation

@aashirvad08

Copy link
Copy Markdown
Contributor

Description

This PR fixes a performance and memory allocation bottleneck when converting an IndexedOptionArray with ListOffsetArray content to Arrow

The fix avoids redundant data duplication by zeroing the starts/stops of masked elements before to_ListOffsetArray64(False), preventing unnecessary allocations while preserving identical Arrow output.

Results

  • ~380× faster execution
  • Peak memory reduced from ~5.15 GB to ~5.9 MB
  • Byte-identical Arrow output

closes #3646

@codecov

codecov Bot commented Jul 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.96%. Comparing base (712dac0) to head (72ee935).
⚠️ Report is 57 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
Files with missing lines Coverage Δ
src/awkward/contents/listarray.py 93.11% <100.00%> (+0.31%) ⬆️

... and 85 files with indirect coverage changes

@ianna ianna changed the title optimize option-type ListArray to Arrow conversion perf: optimize option-type ListArray to Arrow conversion Jul 18, 2026
@github-actions github-actions Bot added the type/perf PR title type: perf (set automatically) label Jul 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type/perf PR title type: perf (set automatically)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Conversion to Arrow of a string array with many nulls allocates excessive memory

1 participant