fix: set_fields for invalid IndexedOption case

[ManasviGoyal]

No description provided.

[zonca]

@ManasviGoyal now the fields are named correctly,
for example:

awkward_array.index_optionA__Zbank1.bank16A__Zname

full layout:

<RecordArray is_tuple='false' len='1'>
    <content index='0' field='index_optionA__Zbank_header'>
        <RecordArray is_tuple='false' len='1'>
            <content index='0' field='bank_hdrA__Zall_bank_size'>
                <NumpyArray dtype='uint32' len='1'>[100]</NumpyArray>
            </content>
            <content index='1' field='bank_hdrA__Zflags'>
                <NumpyArray dtype='uint32' len='1'>[16]</NumpyArray>
            </content>
        </RecordArray>
    </content>
    <content index='1' field='index_optionA__Zbank1'>
        <IndexedOptionArray len='1'>
            <index><Index dtype='int64' len='1'>[-1]</Index></index>
            <content><RecordArray is_tuple='false' len='0'>
                <content index='0' field='bank16A__Zname'>
                    <ListOffsetArray len='0'>
                        <parameter name='__array__'>'string'</parameter>
                        <offsets><Index dtype='int64' len='1'>
                            [0]
                        </Index></offsets>
                        <content><NumpyArray dtype='uint8' len='0'>
                            <parameter name='__array__'>'char'</parameter>
                            []
                        </NumpyArray></content>
                    </ListOffsetArray>
                </content>
                <content index='1' field='bank16A__Ztype'>
                    <NumpyArray dtype='uint16' len='0'>[]</NumpyArray>
                </content>
                <content index='2' field='bank16A__Zsize'>
                    <NumpyArray dtype='uint16' len='0'>[]</NumpyArray>
                </content>
            </RecordArray></content>
        </IndexedOptionArray>
    </content>
    <content index='2' field='index_optionA__Zbank2'>
        <IndexedOptionArray len='1'>
            <index><Index dtype='int64' len='1'>[0]</Index></index>
            <content><RecordArray is_tuple='false' len='1'>
                <content index='0' field='bank32A__Zname'>
                    <ListOffsetArray len='1'>
                        <parameter name='__array__'>'string'</parameter>
                        <offsets><Index dtype='int64' len='2'>[0 4]</Index></offsets>
                        <content><NumpyArray dtype='uint8' len='4'>
                            <parameter name='__array__'>'char'</parameter>
                            [ 66  97 110 107]
                        </NumpyArray></content>
                    </ListOffsetArray>
                </content>
                <content index='1' field='bank32A__Ztype'>
                    <NumpyArray dtype='uint32' len='1'>[3]</NumpyArray>
                </content>
                <content index='2' field='bank32A__Zsize'>
                    <NumpyArray dtype='uint32' len='1'>[2]</NumpyArray>
                </content>
            </RecordArray></content>
        </IndexedOptionArray>
    </content>
</RecordArray>

However the types seem malformed, for example this shows a ?:

In [15]: awkward_array.index_optionA__Zbank1.bank16A__Zname
Out[15]: <Array [None] type='1 * ?string'>

and to_numpy() fails with:

AssertionError: unrecognized dtype: [('bank16A__Zname', '<U1'), ('bank16A__Ztype', '<u2'), ('bank16A__Zsize', '<u2')]

[zonca]

In [23]: awkward_array.index_optionA__Zbank1.bank16A__Zname.layout
Out[23]: 
<IndexedOptionArray len='1'>
    <index><Index dtype='int64' len='1'>[-1]</Index></index>
    <content><ListOffsetArray len='0'>
        <parameter name='__array__'>'string'</parameter>
        <offsets><Index dtype='int64' len='1'>
            [0]
        </Index></offsets>
        <content><NumpyArray dtype='uint8' len='0'>
            <parameter name='__array__'>'char'</parameter>
            []
        </NumpyArray></content>
    </ListOffsetArray></content>
</IndexedOptionArray>

it looks like there is a ListOffsetArray which is of length 0 but has an offset which has length 1, could that cause this problem? @ManasviGoyal @jpivarski

[ManasviGoyal]

In [23]: awkward_array.index_optionA__Zbank1.bank16A__Zname.layout
Out[23]: 
<IndexedOptionArray len='1'>
    <index><Index dtype='int64' len='1'>[-1]</Index></index>
    <content><ListOffsetArray len='0'>
        <parameter name='__array__'>'string'</parameter>
        <offsets><Index dtype='int64' len='1'>
            [0]
        </Index></offsets>
        <content><NumpyArray dtype='uint8' len='0'>
            <parameter name='__array__'>'char'</parameter>
            []
        </NumpyArray></content>
    </ListOffsetArray></content>
</IndexedOptionArray>

it looks like there is a ListOffsetArray which is of length 0 but has an offset which has length 1, could that cause this problem? @ManasviGoyal @jpivarski

I think the offset length is always 1 as the first 0 is added by default

[zonca]

and to_numpy() fails with:

AssertionError: unrecognized dtype: [('bank16A__Zname', '<U1'), ('bank16A__Ztype', '<u2'), ('bank16A__Zsize', '<u2')]

this error is fixed by scikit-hep/awkward#3060,
however I think there might be something wrong with the fill_value:

In [4]: awkward_array.index_optionA__Zbank1.to_numpy()
Out[4]: 
masked_array(data=[(--, --, --)],
             mask=[( True,  True,  True)],
       fill_value=('N', 16959, 16959),
            dtype=[('bank16A__Zname', '<U1'), ('bank16A__Ztype', '<u2'), ('bank16A__Zsize', '<u2')])

In [5]: awkward_array.index_optionA__Zbank1.bank16A__Ztype.to_numpy()
Out[5]: 
masked_array(data=[--],
             mask=[ True],
       fill_value=999999,
            dtype=uint16)

In [8]: awkward_array.index_optionA__Zbank1.to_numpy()["bank16A__Ztype"]
Out[8]: 
masked_array(data=[--],
             mask=[ True],
       fill_value=16959,
            dtype=uint16)

In [6]: awkward_array.index_optionA__Zbank1.bank16A__Zname.to_numpy()
Out[6]: 
masked_array(data=[--],
             mask=[ True],
       fill_value='N/A',
            dtype='<U1')

In [7]: awkward_array.index_optionA__Zbank1.to_numpy()["bank16A__Zname"]
Out[7]: 
masked_array(data=[--],
             mask=[ True],
       fill_value='N',
            dtype='<U1')

[zonca]

actually if I create a similar array in pure numpy:

In [18]: np.ma.zeros(2, dtype = [('string', '<U1'), ('int', np.uint16)])
Out[18]: 
masked_array(data=[('', 0), ('', 0)],
             mask=[(False, False), (False, False)],
       fill_value=('N', 16959),
            dtype=[('string', '<U1'), ('int', '<u2')])

it looks like the structured array fill values agree with the numpy defaults, it is the standard ones which are different.

Actually nevermind, numpy does exactly the same:

In [22]: np.ma.zeros(2, dtype="<U1")
Out[22]: 
masked_array(data=['', ''],
             mask=False,
       fill_value='N/A',
            dtype='<U1')

In [23]: np.ma.zeros(2, dtype=np.uint16)
Out[23]: 
masked_array(data=[0, 0],
             mask=False,
       fill_value=999999,
            dtype=uint16)

however:

In [24]: np.ma.zeros(2, dtype = [('string', '<U1'), ('int', np.uint16)])["string"]
Out[24]: 
masked_array(data=['', ''],
             mask=[False, False],
       fill_value='N',
            dtype='<U1')

In [25]: np.ma.zeros(2, dtype = [('string', '<U1'), ('int', np.uint16)])["int"]
Out[25]: 
masked_array(data=[0, 0],
             mask=[False, False],
       fill_value=16959,
            dtype=uint16)

@jpivarski given numpy does the same I think this behaviour is ok, do you agree?

[zonca]

@ManasviGoyal I think you can merge this PR

[zonca]

ok, thanks @ManasviGoyal, this is good to merge