import from the new location of normalize_axis_index if possible

[keewis]

Another one of the numpy=2.0 fixes, this time numpy.core.multiarray.normalize_axis_index has been moved to numpy.lib.array_utils (and apparently this is the first time it has been officially exposed as public API).

Since as far as I remember numpy is working on removing numpy.core entirely, we might also want to change our usage of defchararray (in the formatting tests). Not sure how, though.

• Towards ⚠️ Nightly upstream-dev CI failed ⚠️ #8091

[keewis]

it appears we have an issue with cftime being built against numpy<2.0 but used with numpy>=2.0. Not sure if we can do anything about that, as cftime build-depends on oldest-supported-numpy, which for python=3.11 pins numpy=1.23.2.

numpy helpfully prints that we should either use wheels built against numpy>=2.0 or disable build isolation when having pip build the wheels from source. In the most recent commit I tried to do the latter, but it didn't work. If anyone has ideas how to fix this, I'm all ears (and @pydata/xarray, feel free to push directly to this branch)

[keewis]

it appears the reason for the failing iris tests is cf_units and the ones for the failing backends tests are netcdf4 and h5py which would all need to be built with numpy>=2.0.0.dev0.

Does anyone have any ideas how to fix that? Otherwise we'd have to live with our upstream-dev CI failing until compatible wheels appear on conda-forge.

[dcherian]

he ones for the failing backends tests are netcdf4 and h5py which would all need to be built with numpy>=2.0.0.dev0.

I doubt these will move before a numpy release candidate.

[keewis]

I guess that means we have two choices:

1. ignore / skip the failing tests until these packages are updated
2. build ourselves in CI

where 2 would mean that we'd have to deal with compiling all three packages (plus make sure the cftime build from this PR works correctly)

[mathause]

For 2. the numpy recommendations seems to be to pass --no-build-isolation to pip but then you have to manually add all build dependencies and it does not correctly set the version number, which causes other problems. And there seems to be no way to force build dependencies (or their version) directly from pip. So I gave up on trying to get the upstream CI to work (for regionmask, but I guess this also applies here, unless I misunderstand something).

[dcherian]

We could also run two configs: with numpy dev, without hdf,netcdf; and one with latest released numpy and dev netcdf, hdf5.

[keewis]

we don't install dev netcdf4 or hdf5 / h5py (as far as I understand it, h5netcdf is pure-python), yet, so unless we change that it would become just "remove netcdf4 and hdf5 until they support numpy dev", which is much easier

[dcherian]

Works for me, since we know numpy is making major changes.

[keewis]

it appears the recent commits fix most of the issues we had with the incompatibility, and we're down to ~15 failing tests now. Thus, I'll mark this as ready and I'll have this PR close the current nightly failures issue, since that has been lingering for way too long.

Edit: The failing doctest requires another warning ignore in xarray.tests

[keewis]

I'll note that flox appears to be using several functions from numpy.core, as well, so we'll have to update that, too.

[keewis]

mypy needs some work (probably ignores?), but otherwise this should be ready

[Illviljan]

xarray/core/nputils.py:15: error: Cannot find implementation or library stub for module named "numpy.lib.array_utils"  [import-not-found]
xarray/core/duck_array_ops.py:43: error: Cannot find implementation or library stub for module named "numpy.lib.array_utils"  [import-not-found]
xarray/core/duck_array_ops.py:43: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports

Numpy has these odd errors sometimes, I haven't fully figured them out yet. Ignore if rephrasing the code doesn't work either.

[dcherian]

Thanks @keewis !