Reshape codec

[jbms]

No description provided.

[jbms]

@ldeakan this is an alternative to the previously discussed squeeze that can accomplish the same thing but is more flexible. What do you think?

It should provide a lot of flexibility for using image codecs and zfp with zarr arrays containing an arbitrary number of dimensions.

[normanrz]

Tagging @LDeakin again, because the linking didn't seem to work.

[LDeakin]

It looks pretty reasonable to me, but I could give you a more definitive review if I get around to implementing it. @jbms do you intend to support this in tensorstore / neuroglancer?

@normanrz Should extensions have an implementation before they get merged in this repo?

[jbms]

I do intend to implement it in tensorstore and neuroglancer. I've started the tensorstore implementation.

There are some tradeoffs I made in designing this:

• Allowing dimensions to be specified indirectly (i.e. as the product of one or more input dimensions, or -1 to mean all remaining) is critical to allow it to work with variable chunking. It adds a bit of complexity in resolving the shape but it doesn't affect anything else.
• I think it will often be useful to combine this with transpose. If all elements of the shape had to be specified as an array of input dims (no explicit size or -1) then it would be natural to allow this codec to also perform transposing. But we already have transpose for transposing and then it wouldn't support cases that require an explicit size for some dimension (though I can't really think of a lot of clear use cases for that).
• For implementations like zarrs that always use a fixed c/Fortran memory layout non-partial encoding and decoding should be trivial. On the other hand, when combined with transpose there may be unnecessary copying. When allowing arbitrary strided layouts as in tensorstore (and I think zarr-python), the non-partial encoding and decoding is more complicated -- some cases can be handled without copying, some cases can't.

[jbms]

I will indeed implement it before switching to non-draft but also wanted to get feedback on it.

[normanrz]

@normanrz Should extensions have an implementation before they get merged in this repo?

There is no strict requirement to have an implementation when registering the name. It is recommended, though.

[LDeakin]

Looks good! Just some minor suggestions.

I implemented it in zarrs: zarrs/zarrs#206

[LDeakin]

It'd be good to demo high-dimensional zfp since it is a motivating use case for this codec

[clbarnes]

this is an alternative to the previously discussed squeeze that can accomplish the same thing but is more flexible

Is there a use case for that flexibility? Array-to-bytes codecs already handle unravelling ND arrays to 1D; I'm struggling to think of any other time you would want to reshape, other than adding/ dropping singleton dimensions. I may have missed the discussions around the squeeze codec - who could use reshaping who couldn't use squeeze?

[jbms]

this is an alternative to the previously discussed squeeze that can accomplish the same thing but is more flexible

Is there a use case for that flexibility? Array-to-bytes codecs already handle unravelling ND arrays to 1D; I'm struggling to think of any other time you would want to reshape, other than adding/ dropping singleton dimensions. I may have missed the discussions around the squeeze codec - who could use reshaping who couldn't use squeeze?

Image formats generally support only 2d images and impose certain constraints on the number of channels. In order to store e.g. a 3d chunk as a 2d image you need to do more than drop singleton dimensions, e.g. turn a [64, 64, 64] xyz array into a [64 * 64, 64] array.

[clbarnes]

That would still end up putting unrelated rows semantically next to each other, wouldn't it? I think that could cause weird artefacts with lossy codecs if you blurred the bottom row of one z-slice in with the top row of the next z-slice just because the reshape rearranged it to be below rather than behind. Not an issue with lossless codecs, of course.

I was envisioning using the sharding codec to create singleton dimensions and then squeeze inside the shard's inner chunks to get rid of them. I suppose this might an issue of multiple layers of sharding, if you also wanted to use shards for inode optimisation; theoretically possible but almost certainly not very well-performing.

The squeeze approach means you don't need to care about the current size of any given dimension, only whether it's singleton or not. I think this would allow squeezes to be compatible with partial IO, where reshapes are not.

[jbms]

That would still end up putting unrelated rows semantically next to each other, wouldn't it? I think that could cause weird artefacts with lossy codecs if you blurred the bottom row of one z-slice in with the top row of the next z-slice just because the reshape rearranged it to be below rather than behind. Not an issue with lossless codecs, of course.

Yes but in practice it still tends to work fine and is useful.

JPEG uses an 8x8 block size so if the y dimension is divisible by 8 then you aren't actually combining unrelated rows at all. And if consecutive z sections are similar then compressing them together is likely to be more efficient than as separate JPEG images as would happen if you use sharding.

I was envisioning using the sharding codec to create singleton dimensions and then squeeze inside the shard's inner chunks to get rid of them. I suppose this might an issue of multiple layers of sharding, if you also wanted to use shards for inode optimisation; theoretically possible but almost certainly not very well-performing.

The squeeze approach means you don't need to care about the current size of any given dimension, only whether it's singleton or not. I think this would allow squeezes to be compatible with partial IO, where reshapes are not.

Partial I/O is still possible with reshape, just more complicated and with more restrictions. But in general, the cases where I imagine reshape to be most useful, e.g. image codecs, wouldn't really support partial I/O anyway. (TIFF allows random access to individual tiles, but you may as well just use sharding rather than tiled TIFF support.)

[jbms]

There are also various reasonable options for an implementation to take, including having one path for certain cases, like the squeeze-only case, that may be faster and/or support extra functionality like partial I/O, and then a fallback path for the general case.

[clbarnes]

Thank you, that's helpful! Is the use case for explicit sizes that you can split a large dimension into smaller chunks e.g. to unpack a 6-channel image into two adjacent 3-channel images?

I'm sure some of these questions are obvious to people with particular use cases; I think it would be helpful to highlight a use case and any recommendations on usage constraints for each feature in a non-normative section of the proposal.

[jbms]

Yes, explicit sizes allow for more flexibility in order to satisfy any necessary constraints of the format, and also allow reshape to represent its own inverse operation, which seemed like a nice property to have.

[clbarnes]

I'm convinced, and have closed the singleton_dim codec PR.

Also, here's a python impl: https://github.com/clbarnes/zarr-python-reshape . That's two - what's the next step for making this PR reviewable?

[psobolewskiPhD]

Bumping this. I have zarr v3 files that have 2D RGB jpeg chunks (via imagecodecs). The shape is not compatible with ome-zarr requirements, so I'm looking for possible solutions. This looks like one, the other being looser ome-zarr spec, which I guess has stalled too....

[clbarnes]

This looks like one, the other being looser ome-zarr spec, which I guess has stalled too....

Is this something which would be addressed by RFC-3 More Dimensions for Thee? It's on the road map, we're just pushing to get RFC-5 Coordinate Transformations in first, which is in the next version of the OME-Zarr spec, currently in release candidate phase

[psobolewskiPhD]

Yes, RFC3 would address this I think, I wasn't clear where it stood since it doesn't appear to be part of 0.6. But yes, because the chunks are YXC/XYC (spatial, spatial, channel), the relaxed requirements should make that compatible.

[clbarnes]

The rubber's meeting the road on this with #66 - if we can establish a reshape codec, then image codecs can all use it, rather than different image codecs encoding or implying their own internal reshapes as the linked does.

@jbms is there anything else you need here? If you're short of time and would be open to someone else being involved in ownership to help get it over the line, I'm happy to do so.

[jbms]

I'm happy to have this submitted -- was just waiting for more feedback and implementation experience. Maybe we have enough now though

[normanrz]

I'll go ahead and merge this later today. We recently added the requirement that extensions need to have at least one implementation to be merged. I guess that has happened in zarrs/zarrs#206.

[clbarnes]

There's an implementation for zarr-python too: https://github.com/clbarnes/zarr-python-reshape