Compatibility policy for zarr-python v3

[TomAugspurger]

Apologies if this has been discussed already, but is there policy written down anywhere with respect to code compatibility written against zarr 2.x? Concretely, I wonder if making this snippet using xarray is a design goal of zarr-python v3?

import xarray as xr
import pathlib
import shutil

p = pathlib.Path("out.zarr")
shutil.rmtree(p, ignore_errors=True)
ds = xr.tutorial.open_dataset("air_temperature")
ds.to_zarr(p)

That runs fine with zarr 2.18.2; Should it work with zarr-python 3.x without modification to xarray? To give you an idea of the kinds of changes needed to make this work:

• store is passed as a positional arg, while it's keyword only on v3 (scikit-learn had a nice decorator for deprecating positional arguments, so if we want store to be keyword only we can deprecate it gracefully without too much effort)
• xarray uses Attributes.put(dict). Easy to alias to .update (with a deprecation if we want)
• xarray accesses various attributes from the Group object (store, read_only, synchronizer). No idea if these still make sense in the v3 codebase.
• Group.create previously took some keyword arguments (shape, dtype, fill_value, chunks) that seem more like Array arguments. Reading through zarr.v2.Group.create, I guess that creates an Array in the group, rather than a new group. Is that deliberate?

There's some additional examples of this in some issues (#2023, #2021)

One point worth discussing: is there value in code-level compatibility if zarr-python 3.x is (presumably) writing zarr-v3 by default? IMO yes, but perhaps not.

[d-v-b]

One point worth discussing: is there value in code-level compatibility if zarr-python 3.x is (presumably) writing zarr-v3 by default? IMO yes, but perhaps not.

Short answer, if we wanted to preserve code-level compatibility, we would be targeting a 2.x release and not a 3.x release.

The longer answer is that supporting zarr version 2 and zarr version 3 and preserving code compatibility with zarr-python v2 (which closely models zarr version 2) is not an easy path. So far, zarr-python 3.x has optimized for supporting zarr version 2 and zarr version 3, at the cost of changing zarr-python 2.x APIs. I think these are the right priorities (but others might disagree). Keep in mind that zarr v3 is a different format from zarr v2. Expecting old APIs to work for a new format seems like a stretch.

That doesn't mean we should wantonly break stuff, and we are open to a variety of solutions for minimizing the changes. The simplest thing is for users who depend on zarr-python 2.x APIs to pin their dependencies to that version of zarr-python. More complicated solutions could be including all of zarr-python 2.x inside a v2 namespace in zarr-python 3.x, or publishing zarr-python 2.x as a stand-alone zarr2 package, etc.

[TomAugspurger]

It's an interesting thought exercise for what has a longer tail: software or data files? :)

Overall I think I agree: Prioritize APIs that make sense for Zarr v3, but don't wantonly break stuff. To make this a bit more concrete I published #2098 as a draft of what it takes to get xarray.Dataset.to_zarr working (probably, I haven't verified stuff 100%).

[jhamman]

I have created #2102 to help address some of these questions. Contributions / feedback on that PR is welcome.