Skip to content

fix: add missing .attrs support in parquet serialisation#4079

Merged
ianna merged 11 commits into
scikit-hep:mainfrom
albert-physics:albert-physics/parquet-support-attrs-serialisation
Jun 10, 2026
Merged

fix: add missing .attrs support in parquet serialisation#4079
ianna merged 11 commits into
scikit-hep:mainfrom
albert-physics:albert-physics/parquet-support-attrs-serialisation

Conversation

@albert-physics

Copy link
Copy Markdown
Contributor

Makes parquet serialisation support reading and writing the .attrs of arrays. This method should also read .attrs written by pandas though has an implicit casting from dict to Attrs which only allows string keys. Tests have been written to test both awkward attrs reading and writing and pandas attrs reading.

This is connected to but does not close #3278 as this does not add support to arrow

@codecov

codecov Bot commented Jun 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.32%. Comparing base (ebfa120) to head (aae47a7).

Additional details and impacted files
Files with missing lines Coverage Δ
src/awkward/operations/ak_from_parquet.py 93.86% <100.00%> (+0.44%) ⬆️
src/awkward/operations/ak_to_parquet.py 65.71% <100.00%> (+1.53%) ⬆️

... and 1 file with indirect coverage changes

@ianna ianna left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@albert-physics - Thanks for contributing! Please update the PR title -- please check the following guidelines:

Available types:

  • feat: A new feature
  • fix: A bug fix
  • docs: Documentation only changes
  • style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
  • refactor: A code change that neither fixes a bug nor adds a feature
  • perf: A code change that improves performance
  • test: Adding missing tests or correcting existing tests
  • build: Changes that affect the build system or external dependencies (example scopes: gulp, broccoli, npm)
  • ci: Changes to our CI configuration files and scripts (example scopes: Travis, Circle, BrowserStack, SauceLabs)
  • chore: Other changes that don't modify src or test files
  • revert: Reverts a previous commit

@albert-physics albert-physics changed the title Albert physics/parquet support attrs serialisation fix: add missing .attrs support in parquet serialisation Jun 9, 2026
@albert-physics

albert-physics commented Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

Fixed the title, I thought it would pick up the title from the last commit name. I have listed this as "fix" mainly because the linked issue referred to it as a bug which required fixing opposed to a new feature.

I have been performing tests locally so am slightly surprised by the failures in the CI. These errors all seem to be from to_parquet_row_groups which handles a list of arrays opposed to arrays themselves. I have added a hasattr(array, "attrs") and array.attrs check to only try to read the attrs if such a variable exists

@albert-physics

Copy link
Copy Markdown
Contributor Author

Some CI tests use an old version of pandas. I have updated the test for reading .attrs from parquet files written with pandas 2.1.0+.

After this is looks like all checks pass

@ianna ianna left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@albert-physics — thanks for implementing this! It looks great.

@ianna
ianna merged commit aa02246 into scikit-hep:main Jun 10, 2026
38 checks passed
@ianna

ianna commented Jun 10, 2026

Copy link
Copy Markdown
Member

@all-contributors please add @albert-physics for code

@TaiSakuma

Copy link
Copy Markdown
Member

The PR description includes the phrase "not close 3278". But I think that GitHub ignored "not" and still closed the issue. I'll reopen the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type/fix PR title type: fix (set automatically)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

attrs are not preserved in serializations through Arrow or Parquet

3 participants