test: add property-based roundtrip test for to_parquet/from_parquet

[TaiSakuma]

This adds a Hypothesis roundtrip test for ak.to_parquet / ak.from_parquet, the third member of the roundtrip property-test family after test_to_from_buffers.py and #4215.

The tests write to fsspec's in-memory filesystem (memory://), so no disk I/O is involved; an example takes about 1 ms. Three tests:

test_roundtrip generates arrays without option types and asserts strict ak.array_equal after the roundtrip.

test_roundtrip_masked enables option types and compares with same_content_types=False, because from_parquet returns every option layout as BitMaskedArray (or UnmaskedArray). It sets allow_regular=False because an option directly over a fixed-size list cannot be written (ARROW-14547).

test_roundtrip_attrs checks that JSON-compatible attrs survive the roundtrip (follow-up coverage for the recent attrs work, #4079/#4209/#4210); keys starting with @ are excluded as intentionally transient.

Generation is restricted to what the parquet path can represent:

• SUPPORTED_PARQUET_DTYPES: complex is excluded (pyarrow cannot write it); datetime64 only ns (coarser units are unsupported or coerced, D corrupts values, and ms/us crash from_parquet when row-group statistics contain NaT or out-of-range values); timedelta64 only s/ms/us/ns.
• allow_union=False (pyarrow cannot write unions to parquet) and allow_empty=False (a bare unknown type cannot be written; a nested one reads back as an option).
• A parquet_writable predicate excludes layouts that the conversion currently mishandles: zero-field records, size-0 RegularArray (write crash), IndexedOptionArray over zero-length content (write crash), a top-level record whose only field is named "" (collides with the anonymous column used for non-record arrays), option-of-record with all-option fields (outermost struct validity is not stored), and nullable var-length lists whose offsets do not start at zero (silent data corruption: every list is shifted by offsets[0]). The predicate rejects about 6% of generated examples.

Four of these exclusions are pre-existing bugs found by this test during development (the datetime64[D] value corruption, the row-group-statistics OverflowError, the zero-length-content write crash, and the offsets-shift corruption). I will file them as separate issues and then update the corresponding comments in this test to cite the issue numbers.

Verified: 200 examples per test (default profile, ~4 s), 10,000 examples per test (nightly profile, ~2.5 min), and pre-commit clean.

🤖 Generated with Claude Code

[codecov]

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
6572	1	6571	34

View the top 1 failed test(s) by shortest run time

tests/properties/operations/test_to_from_parquet.py::test_roundtrip

Stack Traces | 17.6s run time

@no_deadline
>   @given(
        a=st_ak.constructors.arrays(
            dtypes=parquet_dtypes(),
            # a bare unknown type cannot be written ("A null type field may
            # not be non-nullable"), and one nested in a record or regular
            # list reads back as an option
            allow_empty=False,
            # pyarrow cannot write unions to parquet
            allow_union=False,
            allow_indexed_option=False,
            allow_byte_masked=False,
            allow_bit_masked=False,
            allow_unmasked=False,
        ).filter(parquet_writable)
    )

f          = <function given.<locals>.run_test_as_given.<locals>.wrapped_test at 0x146003c8feb0>

.../properties/operations/test_to_from_parquet.py:117: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../properties/operations/test_to_from_parquet.py:135: in test_roundtrip
    ak.to_parquet(a, path)
        a          = <Array [] type='0 * float128'>
        path       = 'memory:.../test-to-from-parquet/22411622082368.parquet'
.../test-env/lib/python3.10....../site-packages/awkward/_dispatch.py:41: in dispatch
    with OperationErrorContext(name, args, kwargs):
        args       = (<Array [] type='0 * float128'>,
 'memory:.../test-to-from-parquet/22411622082368.parquet')
        array_like = <Array [] type='0 * float128'>
        array_likes = (<Array [] type='0 * float128'>,)
        dispatch   = <function to_parquet at 0x146009731870>
        func       = <function to_parquet at 0x1460097317e0>
        gen_or_result = <generator object to_parquet at 0x145a2ab8c890>
        kwargs     = {}
        name       = 'ak.to_parquet'
.../test-env/lib/python3.10.../site-packages/awkward/_errors.py:80: in __exit__
    raise self.decorate_exception(exception_type, exception_value)
        exception_type = <class 'pyarrow.lib.ArrowNotImplementedError'>
        exception_value = ArrowNotImplementedError('Unsupported numpy type 13')
        self       = <awkward._errors.OperationErrorContext object at 0x145a30084790>
        traceback  = <traceback object at 0x145a2a903100>
.../test-env/lib/python3.10....../site-packages/awkward/_dispatch.py:67: in dispatch
    next(gen_or_result)
        args       = (<Array [] type='0 * float128'>,
 'memory:.../test-to-from-parquet/22411622082368.parquet')
        array_like = <Array [] type='0 * float128'>
        array_likes = (<Array [] type='0 * float128'>,)
        dispatch   = <function to_parquet at 0x146009731870>
        func       = <function to_parquet at 0x1460097317e0>
        gen_or_result = <generator object to_parquet at 0x145a2ab8c890>
        kwargs     = {}
        name       = 'ak.to_parquet'
.../test-env/lib/python3.10.../awkward/operations/ak_to_parquet.py:196: in to_parquet
    return _impl(
        array      = <Array [] type='0 * float128'>
        bytestring_to32 = True
        categorical_as_dictionary = False
        compression = 'zstd'
        compression_level = None
        count_nulls = True
        data_page_size = None
        destination = 'memory:.../test-to-from-parquet/22411622082368.parquet'
        emptyarray_to = None
        extensionarray = True
        list_to32  = False
        parquet_byte_stream_split = False
        parquet_coerce_timestamps = None
        parquet_compliant_nested = False
        parquet_dictionary_encoding = False
        parquet_extra_options = None
        parquet_flavor = None
        parquet_metadata_statistics = True
        parquet_old_int96_timestamps = None
        parquet_page_version = '1.0'
        parquet_version = '2.6'
        row_group_size = None
        storage_options = None
        string_to32 = True
.../test-env/lib/python3.10.../awkward/operations/ak_to_parquet.py:282: in _impl
    layout, table = get_layout_and_table(data)
        array      = <Array [] type='0 * float128'>
        bytestring_to32 = True
        categorical_as_dictionary = False
        compression = 'zstd'
        compression_level = None
        count_nulls = True
        data       = <Array [] type='0 * float128'>
        data_page_size = None
        destination = 'memory:.../test-to-from-parquet/22411622082368.parquet'
        emptyarray_to = None
        extensionarray = True
        get_layout_and_table = <function _impl.<locals>.get_layout_and_table at 0x145a2ab43d90>
        list_to32  = False
        parquet_byte_stream_split = False
        parquet_coerce_timestamps = None
        parquet_compliant_nested = False
        parquet_dictionary_encoding = False
        parquet_extra_options = None
        parquet_flavor = None
        parquet_metadata_statistics = True
        parquet_old_int96_timestamps = None
        parquet_page_version = '1.0'
        parquet_version = '2.6'
        pyarrow_parquet = <module 'pyarrow.parquet' from '.../test-env/lib/python3.10.../pyarrow/parquet/__init__.py'>
        row_group_size = None
        storage_options = None
        string_to32 = True
        write_iteratively = False
.../test-env/lib/python3.10.../awkward/operations/ak_to_parquet.py:269: in get_layout_and_table
    table = ak.operations.ak_to_arrow_table._impl(
        bytestring_to32 = True
        categorical_as_dictionary = False
        count_nulls = True
        emptyarray_to = None
        extensionarray = True
        layout     = <NumpyArray dtype='float128' len='0'>[]</NumpyArray>
        list_to32  = False
        string_to32 = True
        x          = <Array [] type='0 * float128'>
.../test-env/lib/python3.10.../awkward/operations/ak_to_arrow_table.py:167: in _impl
    layout.to_arrow(
        array      = <NumpyArray dtype='float128' len='0'>[]</NumpyArray>
        arrow_arrays = []
        arrow_fields = []
        bytestring_to32 = True
        categorical_as_dictionary = False
        count_nulls = True
        direct_Content_subclass = <function direct_Content_subclass at 0x1460096d5360>
        emptyarray_to = None
        extensionarray = True
        layout     = <NumpyArray dtype='float128' len='0'>[]</NumpyArray>
        list_to32  = False
        pyarrow    = <module 'pyarrow' from '.../test-env/lib/python3.10........./site-packages/pyarrow/__init__.py'>
        record_is_scalar = False
        schema_parameters = None
        string_to32 = True
        wrapper_layouts = [<NumpyArray dtype='float128' len='0'>[]</NumpyArray>]
.../test-env/lib/python3.10.../awkward/contents/content.py:1093: in to_arrow
    return self._to_arrow(
        awkward    = <module 'awkward' from '.../test-env/lib/python3.10.../site-packages/awkward/__init__.py'>
        bytestring_to32 = True
        categorical_as_dictionary = False
        count_nulls = True
        emptyarray_to = None
        extensionarray = True
        list_to32  = False
        pyarrow    = <module 'pyarrow' from '.../test-env/lib/python3.10........./site-packages/pyarrow/__init__.py'>
        record_is_scalar = False
        self       = <NumpyArray dtype='float128' len='0'>[]</NumpyArray>
        string_to32 = True
.../test-env/lib/python3.10.../awkward/contents/numpyarray.py:1161: in _to_arrow
    storage_type = pyarrow.from_numpy_dtype(nparray.dtype)
        length     = 0
        mask_node  = None
        nparray    = array([], dtype=float128)
        options    = {'bytestring_to32': True,
 'categorical_as_dictionary': False,
 'count_nulls': True,
 'emptyarray_to': None,
 'extensionarray': True,
 'list_to32': False,
 'record_is_scalar': False,
 'string_to32': True}
        pyarrow    = <module 'pyarrow' from '.../test-env/lib/python3.10........./site-packages/pyarrow/__init__.py'>
        self       = <NumpyArray dtype='float128' len='0'>[]</NumpyArray>
        validbytes = None
pyarrow/types.pxi:5979: in pyarrow.lib.from_numpy_dtype
    ???
pyarrow/error.pxi:155: in pyarrow.lib.pyarrow_internal_check_status
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   pyarrow.lib.ArrowNotImplementedError: Unsupported numpy type 13
E   
E   
E   See if this has been reported at https://github..../scikit-hep/awkward/issues
E   Falsifying example: test_roundtrip(
E       a=<Array [] type='0 * float128'>,
E   )
E   
E   You can reproduce this example by temporarily adding @reproduce_failure('6.158.1', b'AABBAkENAAA=') as a decorator on your test case


pyarrow/error.pxi:92: ArrowNotImplementedError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

[github-actions]

The documentation preview is ready to be viewed at http://preview.awkward-array.org.s3-website.us-east-1.amazonaws.com/PR4218

[ianna]

@TaiSakuma — this is fantastic! More bugs revealed and fixed. Thank you!