NumPy 2.0 support #38
Comments
|
Thanks @jakirkham ! I think this is a great approach. I looked through this list of dependencies today and can't think of any others or a different approach to identify them. And I checked the statuses of all the not-yet-released ones and don't see any changes. |
|
Went through the project list again earlier today and also now Main changes were the GDAL release went out Also Numba RCs are available Dask may work with NumPy 2, but needs reconfirmation Added a better issue link for TensorFlow Tried to also split apart when upstream has fixes (like Dask or XGBoost) from whether they are released. Hopefully that gives a bit more visibility into the state of NumPy 2 support |
|
|
|
Thanks Ralf! |
|
Thanks for the reminder! 🙏 Have refreshed the table above |
|
Numba 0.60.0, Xarray 2024.6.0, and CuPy 13.2.0 were all released. So looks like things are mostly good here (a few rough edges left). |
|
Imageio and XGBoost were both released as well. |
|
You can cross ImageIO off the list ... we now support numpy v2.0 as of ImageIO v2.34.2 |
|
Thanks all! 🙏 Have refreshed the list Looks like we are down to TensorFlow. This is only needed in some cases. So think it makes sense to start doing this work at this point |
|
It seems xref scikit-learn-contrib/hdbscan#644 (fix, but remaining issue) |
This just fixes a simple error that would appear when run with NumPy 2 based on the ruff check. (This does not mean there are no other issues.) xref: rapidsai/build-planning#38 Authors: - Sebastian Berg (https://github.com/seberg) Approvers: - Ajay Thorve (https://github.com/AjayThorve) URL: #606
|
IIRC hdbscan is only a test dependency, JFYI. |
Splitting out the non API changes from gh-15897, the Scalar API change is required for the tests to pass with NumPy 2, but almost all changes should be relatively straight forward here on their own. (I will add inline comments.) --- This PR does not fix integer comparisons, there are currently no tests that run into these. xref: rapidsai/build-planning#38 Authors: - Sebastian Berg (https://github.com/seberg) Approvers: - Matthew Roeschke (https://github.com/mroeschke) URL: #16141
…#16140) This aligns with NumPy, which deprecated this since a while and raises an error now on NumPy 2, for example for `Scalar(-1, dtype=np.uint8)`. Since it aligns with NumPy, the DeprecationWarning of earlier NumPy versions is inherited for those. This (or similar handling) is required to be compatible with NumPy 2/pandas, since the default needs to be to reject operation when values are out of bounds for e.g. `uint8_series + 1000`, the 1000 should not be silently cast to a `uint8`. --- Split from gh-15897 xref: rapidsai/build-planning#38 Authors: - Sebastian Berg (https://github.com/seberg) Approvers: - Matthew Roeschke (https://github.com/mroeschke) URL: #16140
|
Just making connections here... those |
This applies some smaller NumPy 2 related fixes. With (in progress) cupy 13.2 fixups, the single gpu test suite seems to be doing mostly fine. There is a single test remaining:
```
test_simpl_set.py::test_simplicial_set_embedding
```
is failing with:
```
(Pdb) cp.asarray(cu_embedding)
array([[23067.518, 23067.518],
[17334.559, 17334.559],
[22713.598, 22713.598],
...,
[23238.438, 23238.438],
[25416.912, 25416.912],
[19748.943, 19748.943]], dtype=float32)
```
being completely different from the reference:
```
array([[5.330462 , 4.3419437],
[4.1822557, 5.6225405],
[5.200859 , 4.530094 ],
...,
[4.852359 , 5.0026293],
[5.361374 , 4.1475334],
[4.0259256, 5.7187223]], dtype=float32)
```
And I am not sure why that might be, I will prod it a bit more, but it may need someone who knows the methods to have a look.
One wrinkle is that hdbscan is not yet released for NumPy 2, but I guess that still required even though sklearn has a version?
(Probably, not a big issue, but my fixups scikit-learn-contrib/hdbscan#644 run into some issue even though it doesn't seem NumPy 2 related.)
xref: rapidsai/build-planning#38
Authors:
- Sebastian Berg (https://github.com/seberg)
- https://github.com/jakirkham
- Dante Gama Dessavre (https://github.com/dantegd)
Approvers:
- Kyle Edwards (https://github.com/KyleFromNVIDIA)
- Dante Gama Dessavre (https://github.com/dantegd)
URL: #5954
|
Regarding HDBSCAN, we updated to 0.8.38.post1 yesterday in PR: rapidsai/cuml#5906 This supports NumPy 2 and, as James pointed out, switches to installing prebuilt binaries |
|
@jakirkham @seberg for all the libraries that were blocked waiting on Or do you think that's too aggressive? |
|
The 3 of us discussed James' question offline. We concluded If a user simply installs CuPy, they will NumPy 2.1 + CuPy 13.3.0, which works A user may not run into the issues we have seen with CuPy (and this is true for some RAPIDS libraries as well). For clarity, these are the 2 issues: Also a user may be combining CuPy 13.2.0 with NumPy 1, which also works So we are thinking to leave it as-is. That said, we can always update this based on user feedback |
Part of issue: rapidsai/build-planning#38 Start building `cudf` with `numpy` version `2.0`. This remains compatible with `numpy` version `1.x` and `2.x`. Allows us to test building with `numpy` version `2.0` (and make sure we catch any issues that show up). Also relaxes the `numpy` `1.x` pin. Pulls in the RDFG changes that are rolling out for broader RAPIDS NumPy 2 support. Authors: - https://github.com/jakirkham - Sebastian Berg (https://github.com/seberg) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) - Ray Douglass (https://github.com/raydouglass) - James Lamb (https://github.com/jameslamb) URL: #16300
|
We're pretty close to being able to install all of RAPIDS alongside @hcho3 and I looked through the open issues today, here's the dependency graph of remaining work: ---
title: NumPy 2.0 dependencies
---
flowchart TD
A[fmt/spdlog] --> B[cuspatial]
B --> C[cuxfilter]
B --> D[rapids]
E[rapids-xgboost] --> D
F[cugraph] --> G[cugraph-gnn]
F --> D
G -- (optional for 24.10) --> H[cugraph-pg]
Represented in link-to-ticket form:
And then finally, after that:
I'll start an internal chat thread to try coordinating the work. |
NumPy 2.0 is coming out soon ( numpy/numpy#24300 ). NumPy 2.0.0rc1 packages for conda & wheels came out 2 weeks back ( numpy/numpy#24300 (comment) )
Ecosystem support for NumPy 2.0 is being tracked in issue: numpy/numpy#26191
Also conda-forge is discussing how to support NumPy 2.0: conda-forge/conda-forge.github.io#1997
When building against NumPy 2.0, it is possible with default settings to build packages that are compatible with NumPy 1 & 2. Where NumPy will target the oldest NumPy version that was built for that Python version being targeted
Developed the following list by installing RAPIDS 24.04 and inspecting, which packages used NumPy. Specifically ran the commands below
This generated a list of dependencies. Some of these were RAPIDS packages themselves. So removed those from the list. Also dropped some indirect dependencies of RAPIDS. Admittedly this can get a little subjective. Though tried to capture a sufficiently complete, though not overly detailed, picture
From this, have built the table below
Some versions have questions marks if...
Blank entries mean no information is known about those fields at this time
Note to editors: Also attaching the CSV file used to generate this table (as editing Markdown tables can be tricky 😅). Would suggest making any changes in the CSV file locally (with Excel or other). Then you can use
prettytable(available on PyPI & Conda-forge) to generate Markdown with this code. The resulting content can be copy-pasted above. Can drag and drop the CSV file into this textbox to attach itprettytablecode:The text was updated successfully, but these errors were encountered: