[Python] `FixedShapedTensor` `to_pandas_dtype` is returning `NotImplementedError`

[goutamvenkat-anyscale]

Describe the bug, including details regarding any error messages, version, and platform.

Repro Script:

import pyarrow as pa

print(f"pyarrow {pa.__version__}")

t = pa.fixed_shape_tensor(pa.float32(), [2, 2])
print(f"type: {t!r}")

print("calling t.to_pandas_dtype() ...")
dtype = t.to_pandas_dtype()
print(f"result: {dtype!r}")

Output:

pyarrow 23.0.0
type: FixedShapeTensorType(extension<arrow.fixed_shape_tensor[value_type=float, shape=[2,2]]>)
calling t.to_pandas_dtype() ...
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
Cell In[2], line 9
      6 print(f"type: {t!r}")
      8 print("calling t.to_pandas_dtype() ...")
----> 9 dtype = t.to_pandas_dtype()
     10 print(f"result: {dtype!r}")
NotImplementedError: extension<arrow.fixed_shape_tensor[value_type=float, shape=[2,2]]>

Seems to contradict what's available in the docs: https://arrow.apache.org/docs/python/generated/pyarrow.FixedShapeTensorType.html#pyarrow.FixedShapeTensorType

For context, this method needs to be implemented in order to use table.to_pandas(split_blocks=True) for per-column extension blocks to reduce memory pressure.

Component(s)

Python

[AlenkaF]

Thank you for opening the issue. Ideally, to_pandas_dtype would not be listed in the documentation under any of the canonical extension types, or BaseExtensionType for that matter.

There is a connected issue discussing fallback for to_pandas in case of extension arrays, see: #33134

I am not sure if this discussion should be moved to #33134 but I do think it would be nice to move the discussion forward. Ideas welcome!

cc @jorisvandenbossche