Doc/v3 migration guide

[jhamman]

Over the past few weeks, we've had a number of conversations/questions about the policy for backward compatibility, deprecations, and breaking changes with the upcoming 3.0 release. This doc is meant to help us iterate toward common language. In its initial form, it is not complete.

Goals for the text here:

1. Developers of 2.18 and 3.0 should be able to decide if backward compatibility is a required attribute of a contribution
2. Users of Zarr should be able to understand if their application will be impacted by the upcoming 3.0 release
3. Users of Zarr should be able to make a plan for how they will adapt their usage of Zarr after the release
4. [non-goal] This is not meant to provide a comprehensive listing of the changes to the zarr API

cc @zarr-developers/python-core-devs

[TomAugspurger]

One thing that might be helpful: what's the group's tolerance for either compatibility code or deprecations as a way to ease the transition? It sounds like strict backwards compatibility (perhaps with warnings) isn't a goal. Is there tolerance for things like #2098 (e.g. restore some properties to the Group object, loosen the keyword-only requirement for some functions). Likewise for things like "cleaning up internal and user facing APIs", which could be done with a deprecation warning. Even if there isn't tolerance for backwards compatibility shims that really clash with the V3 spec or the current v3 implementation?

[dstansby]

I left some comments - I'll mark this as request changes since it still needs major additions.

[dstansby]

Suggested change

	- ``numcodecs.*`` will no longer be available in ``zarr.*``. (Suggested action: transition to importing codecs from ``numcodecs`` directly.)
	- ``zarr.numcodecs.*`` will no longer be available. These imports can be replaced by importing ``numcodecs`` directly.

We should be doing more than "suggesting", we should be providing concrete fixes/code updates!

[jhamman]

I wish it were that easy! Unfortunately, the top level namespace is littered with * imports

zarr-python/zarr/__init__.py

Line 2 in e1d98cd

from zarr.codecs import *

zarr-python/zarr/codecs.py

Line 2 in e1d98cd

from numcodecs import *

If there is a reasonable way for us to deprecate folks using these imports, I'm all ears.

[dstansby]

Even if there isn't a reasonable way to deprecate the imports, as a minimum this migration guide should provide a complete list of all imports that are disappearing and how to replace them.

[dstansby]

This needs a clear list of how to update code to adapt to these removals or changes.

[jni]

I just came here following the link from #1849. Currently, this guide feels very incomplete, and since #2182 is now in-flight, I think this guide should become very clear.

From the perspective of a maintainer / close user of a bunch of libraries that depend on zarr but have extremely limited maintainer time, the most important thing I want to understand is how hard it will be to support both zarr 2.18 and zarr 3+ within a single library. I think this is the critical question for a smooth transition, because it is hard for libraries to all migrate at the same time, and you want libraries to be installable together in the same environment — you don't want someone depending on both napari and ome-zarr to face napari requiring zarr>=3 and ome-zarr requiring zarr<3. So many libraries would want to support the subset of zarr that is identical in v3 and v2 until everyone can agree to depend on 3+.

[jhamman]

@jni - thanks for the feedback. I agree this is not ready to ship yet. The main things that we know are changing is the Store API and access to internal APIs (e.g. zarr.core.xxx). Beyond that, the best way for us to fill out the migration guide is to have projects attempt to support v3 and report back. I'm doing that with Dask right now (dask/dask#11388, #2186) and I understand @TomAugspurger has begun the process for Xarray. So an ask for you and the Napari / ome-zarr devs is to try to do this and report back. Beyond that, specific suggestions to this doc are more than welcome (props to @dstansby for his edits already).

[dstansby]

If we want downstream packages to test with version 3.0.0a1, it might be good to do a blog post or add something to the docs explaining how to do that testing, what to look for, and how to provide feedback?