[C++][Python] IPC serialization drops a union's buffers when the union is wrapped in an extension type (invalid data; segfault on read)

[TaiSakuma]

Describe the bug, including details regarding any error messages, version, and platform

Serializing a table whose column is a registered ExtensionType with union storage through Arrow IPC (file or stream format) produces bytes in which the union's buffers are missing. Reading them back yields a table that fails validate(full=True) with Buffer #1 too small, and accessing the values (to_pylist) crashes the process with a segmentation fault. The table validates before writing, and the same union columns without the extension wrapper roundtrip correctly.

import io
import pyarrow as pa
import pyarrow.ipc


class MyExt(pa.ExtensionType):
    def __init__(self, storage_type):
        super().__init__(storage_type, "my.ext")

    def __arrow_ext_serialize__(self):
        return b""

    @classmethod
    def __arrow_ext_deserialize__(cls, storage_type, serialized):
        return cls(storage_type)


storage = pa.UnionArray.from_dense(
    pa.array([0, 1], type=pa.int8()),
    pa.array([0, 0], type=pa.int32()),
    [pa.array([1.5], type=pa.float64()), pa.array(["abc"], type=pa.large_string())],
)
pa.register_extension_type(MyExt(storage.type))
ext = pa.ExtensionArray.from_storage(MyExt(storage.type), storage)
t = pa.table({"x": ext})
t.validate(full=True)  # passes; t.column("x").to_pylist() == [1.5, "abc"]

sink = io.BytesIO()
with pa.ipc.new_file(sink, t.schema) as w:
    w.write_table(t)
t2 = pa.ipc.open_file(io.BytesIO(sink.getvalue())).read_all()

t2.validate(full=True)
# ArrowInvalid: Column 0: In chunk 0: Invalid: Buffer #1 too small in array of
# type extension<my.ext<MyExt>> and length 2: expected at least 2 byte(s), got 0

t2.column("x").to_pylist()
# Segmentation fault (exit code 139)

All eight combinations of {dense, sparse} union × {file, stream} IPC format × {extension-wrapped, bare}:

storage	column type	format	result
dense union	extension	file	ArrowInvalid: Buffer #1 too small …
dense union	extension	stream	ArrowInvalid: Buffer #1 too small …
sparse union	extension	file	ArrowInvalid: Buffer #1 too small …
sparse union	extension	stream	ArrowInvalid: Buffer #1 too small …
dense union	bare	file	OK
dense union	bare	stream	OK
sparse union	bare	file	OK
sparse union	bare	stream	OK

The defect is on the write side: reading the same file in a fresh process in which my.ext is NOT registered falls back to the storage type (a plain dense_union column with ARROW:extension:name in the field metadata) and fails validate(full=True) with the same error, so the union's buffers are absent from the serialized bytes themselves rather than being lost during extension-type reconstruction.

Reproduced with pyarrow 23.0.1 and 25.0.0 (pip wheels), Python 3.10.20, macOS 26.5 arm64. Also observed through pyarrow.feather version 2 files, which use the IPC file format. The commit history of cpp/src/arrow/ipc/ contains no related fix after 25.0.0.

This looks like the same mechanism as #15068 (IPC roundtrip of an extension type with pa.null() storage reconstructs invalid buffers): both null and union are types whose buffer layout differs from the default — null has no buffers and unions have no validity buffer — and in both cases the extension wrapper breaks the IPC buffer accounting for them. Possibly one for the #49963 umbrella.

Context: encountered while property-based-testing the feather I/O of the Awkward Array library (scikit-hep/awkward).

Component(s)

C++, Python

🤖 Generated with Claude Code