more engine environment tricks in preparation for numpy>=2

[keewis]

Turns out pydap also needs to build with numpy>=2. Until it does, we should remove it from the upstream-dev environment. Also, numcodecs build-depends on setuptools-scm.

And finally, the h5py nightlies might support numpy>=2 (h5py>=3.11 supposedly is numpy>=2 compatible), so once again I'll try and see if CI passes.

• towards ⚠️ Nightly upstream-dev CI failed ⚠️ #8844

[keewis]

@kmuehlbauer, I might need some help here... There's two h5netcdf tests that fail because the h5py version from scientific-python-nightly-wheels was not built with ros3-support. The tests are decorated with requires_h5netcdf_ros3, but that simply checks that the h5netcdf version is at least 1.3.0. Do you know if there is a way to skip these tests if h5py does not support ros3? If so, feel free to directly push to the PR (or tell me and I'll make the changes).

Once we merge this will re-enable the h5netcdf tests in the upstream-dev CI.

[kmuehlbauer]

Thanks @keewis for taking care.

Yes, this test was probably created with the assumption of conda-forge environment in mind.

I'll need to look into this how to check for ros3 within h5py.

We might use h5py.get_config(). It should have a ros3 attribute set to True when ros3 is available (ros3=h5py.get_config().ros3).

This is from the top of my head, so please double-check.

[keewis]

(to confirm, is the issue that it can't build with numpy>=2?)

no, just that we don't run upstream-dev pydap in CI. This means that we get conda-forge's version of pydap, which is not built against numpy>=2 (i.e. it is not binary compatible). I did not look into building it in CI.

We might use h5py.get_config(). It should have a ros3 attribute set to True when ros3 is available

Thanks a lot, I'll see what I can do.

[keewis]

that appears to have done it! Which means this is ready for review and merging.

Edit: or maybe not... another condition I got wrong. Should work now, though.

[kmuehlbauer]

@keewis 💯 👍 Great to have h5netcdf upstream testing back!

[keewis]

lots of new upstream-dev failures because of the new internal warning filter. Most of these are resolved in #8939 (so we should try to resolve the typing issues that currently block it), with the remaining one being a warning about deprecating the cross product between 2D vectors (from the xr.cross tests).

Edit: the macos 3.11 appears to be flaky as usual for the macos CI. I'll go ahead and merge anyways.