cast numpy scalars to arrays in as_compatible_data

[keewis]

• Closes Variable may contain numpy scalars with numpy>=2.1 #9399
• Tests added

As mentioned in #9399, numpy recently added __array_namespace__ to scalars, which will cause Variable to wrap the scalar. This also casts numpy scalars to numpy.ndarray if it ever reaches as_compatible_data.

This will most likely also pop up in NamedArray / xarray.namedarray.from_array, so we might have to figure out what to do there. cc @andersy005 and @Illviljan for awareness

[TomNicholas]

This will most likely also pop up in NamedArray / xarray.namedarray.from_array, so we might have to figure out what to do there.

I think the equivalent check is here

xarray/xarray/namedarray/core.py

Line 207 in 19a0428

if isinstance(data, _arrayfunction_or_api):

This does a runtime_checkable check against the _arrayfunction_or_api, protocol type. That protocol includes __array_namespace__ but also includes shape, so if the numpy scalar doesn't have a shape attribute then maybe this will already work correctly?

[keewis]

numpy scalars do have a shape (and ndim / dtype), just like 0d arrays this is (). It feels like numpy scalars have become something like a combination of arrays and python scalars.

[Illviljan]

Returning scalars is not supposed to happen if the array api is respected. Hopefully numpy considers this a a bug too.

import numpy as np
import array_api_strict as xps


a = xps.asarray([1,2.0])
xps.mean(a)
Out[21]: Array(1.5, dtype=array_api_strict.float64)


a = np.asarray([1,2.0])
np.mean(a)
Out[23]: np.float64(1.5)

[keewis]

Returning scalars is not supposed to happen if the array api is respected.

Indeed. However, that is only if you read it as "only arrays can be array API compliant". It appears that numpy went the other way and made numpy scalars behave like 0d arrays, thus becoming array API compliant. In other words, numpy scalars now define ndim, dtype, and shape, as well as __array_namespace__ (and any other required properties / methods I might have forgotten).

[TomNicholas]

thus becoming array API compliant

It surprised me that this is even allowed within the array API. For example

In [1]: import numpy as np

In [2]: s = np.float64(4.1)

In [3]: result = np.broadcast_to(s, (2, 2))

In [4]: result
Out[4]: 
array([[4.1, 4.1],
       [4.1, 4.1]])

In [5]: type(result)
Out[5]: numpy.ndarray

In [6]: type(s)
Out[6]: numpy.float64

means that in the broadcast_to specification in the array API standard then the class represented by the array input type hint going in is different from the class represented by the returned array type hint. BUt it turns out that is actually allowed!

I wonder if there are other places in xarray where we assume that for an array-API-compliant duck type, the type going into an array API function/method is going to be the same as the type coming out (because apparently it doesn't have to be).

[keewis]

@Illviljan, should I try to apply something like this to namedarray, as well? Or do you want to figure out a different way to exclude numpy scalars in namedarray? In any case, feel free to push to this PR if so.

Otherwise I believe this should be ready for a final review?

[shoyer]

This is worth adding a comment, noting that we want to cast numpy scalars to arrays.