`ak.forms.form.index_to_dtype` is probably wrong: should probably be native, not little-endian

[jpivarski]

Version of Awkward Array

HEAD

Description and code to reproduce

Compare

awkward/src/awkward/types/numpytype.py

Lines 67 to 83 in fb245f1

	_primitive_to_dtype_dict = {
	"bool": np.dtype(np.bool_),
	"int8": np.dtype(np.int8),
	"uint8": np.dtype(np.uint8),
	"int16": np.dtype(np.int16),
	"uint16": np.dtype(np.uint16),
	"int32": np.dtype(np.int32),
	"uint32": np.dtype(np.uint32),
	"int64": np.dtype(np.int64),
	"uint64": np.dtype(np.uint64),
	"float32": np.dtype(np.float32),
	"float64": np.dtype(np.float64),
	"complex64": np.dtype(np.complex64),
	"complex128": np.dtype(np.complex128),
	"datetime64": np.dtype(np.datetime64),
	"timedelta64": np.dtype(np.timedelta64),
	}

which sets the dtype for each primitive category to the native-endianness for the machine (np.dtype(np.float64) will be big-endian on big-endian machines and little-endian on little-endian machines) with

awkward/src/awkward/forms/form.py

Lines 375 to 381 in 8285871

	index_to_dtype: Final[dict[str, DType]] = {
	"i8": np.dtype("<i1"),
	"u8": np.dtype("<u1"),
	"i32": np.dtype("<i4"),
	"u32": np.dtype("<u4"),
	"i64": np.dtype("<i8"),
	}

which sets the dtype for each index category to little-endian. This should probably be native-endian, too. I'm 90% sure of it.

The problem is that we'd like to test it before we change it. Is there any way we can test Awkward on a big-endian machine? Maybe in a qemu emulation? (Such a test would probably reveal a lot.)

On the flip side, the fact that it's so hard to find a big-endian machine these days means that this issue would rarely be observed. (Similar to 32-bit testing...)

[ianna]

I have an old PowerPC in the office. I'll check if I can use it to test this.

[ariostas]

I looked into this a bit. The only big endian architecture that has a reasonable level of support by Python is s390x, and it's only at tier 3. There are not even Numpy distributions for s390x on PyPI or conda-forge, so it's safe to assume that key dependencies are, at best, not well tested, and at worst, completely broken. So, in my opinion, it's fine to unapologetically only support little endian.

On the other hand, 32-bit architectures are making a comeback with WASM, but that seems to be working fine.

[agoose77]

I'm pretty sure Jim's right here; we convert between endianness when we invoke to_buffers and from_buffers; the forms should not know about this. It would be good to figure out how to explicitly trigger the bug, which ought to exist 🤔

[ikrommyd]

I tested this on a QEMU s390x machine and indeed this is more than 50% of the problem of #3630. The other percentage is how endianess is (or isn't) handled in to/from buffers and the c++ array builder.

[ianna]

Fixed in #3629