Use HDBSCAN package pin to 0.8.38

[divyegala]

HBSCAN has a breaking change in main branch, which cause wheel nightly tests to fail as cuML picks up HDBSCAN from main branch HEAD. As HDBSCAN also releases on pypi now, we can install release packages instead of installing HDBSCAN from git source.

This PR also updates HBDSCAN to v0.8.38.

[jameslamb]

This is failing because hdbscan pins to cython<3, while most of RAPIDS uses cython>=3.

  The conflict is caused by:
      The user requested cython>=3.0.0
      hdbscan 0.8.36 depends on cython<3 and >=0.27

(build link)

Hopefully we'll be able to resolve that by using the next hdbscan release... since @seberg relaxed that cython constraint for hdbscan in scikit-learn-contrib/hdbscan#644 🎉

[seberg]

Hopefully hdbscan will release soon, otherwise it seems like we may need to restore the pull from master (or commit hash) for both conda and pip for a while.

[trxcllnt]

hdbscan still has cython>=0.27,<3 in its top-level requirements.txt.

This is how cython>=0.27,<3 is ending up in install_requires and should be removed.

[jakirkham]

Also this is still targeting branch-24.06. Think this would need to be redirected to branch-24.10

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[divyegala]

/ok to test

[jakirkham]

Thanks Divye! 🙏

Had a suggestion below

Also can you please update this line?

cuml/conda/recipes/cuml-cpu/meta.yaml

Line 37 in 990db71

- hdbscan<=0.8.30

[divyegala]

/ok to test

[jakirkham]

Thanks Divye! 🙏

[jakirkham]

Are we using Cython in a PyTest plugin somewhere?

Seeing this error on CI

/opt/conda/envs/test/lib/python3.11/site-packages/_pytest/config/__init__.py:331: PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
Plugin: helpconfig, Hook: pytest_cmdline_parse
ModuleNotFoundError: No module named 'Cython'

Edit: Are we trying to install something from source (that uses Cython) at test time?

[seberg]

Are we trying to install something from source (that uses Cython) at test time?

Yeah, the cython coverage needs Cython.

[jakirkham]

Thanks Sebastian! 🙏

That makes much more sense. So the Cython dependency just comes from the fact that cuML itself uses a Cython plugin for measuring test coverage. Looks like this has been around since code coverage began being measured ( #3338 )

Edit: Cython code coverage was introduced in PR: #3111

cuml/python/cuml/.coveragerc

Line 4 in 0fbd919

plugins = Cython.Coverage

Before, this dependency was not expressed in cuML's test requirement. However, as it was being implicitly handled via HDBSCAN, where HDBSCAN had an extraneous runtime dependency on Cython, things just quietly worked

With the recent HDBSCAN update, this extraneous Cython dependency was dropped ( scikit-learn-contrib/hdbscan#624 ). So we no longer get Cython pulled in transitively. Hence the error coming up now

To fix this, think we need to add cython to this list and run rapids-dependency-file-generator to propagate the change

cuml/dependencies.yaml

Lines 464 to 468 in 0fbd919

	test_python:
	common:
	- output_types: [conda, requirements, pyproject]
	packages:
	- dask-ml

[jakirkham]

Looks like there was a network failure in one CI job:

[nltk_data] Downloading package treebank to /github/home/nltk_data...
[nltk_data] Error downloading 'treebank' from
[nltk_data]     <https://raw.githubusercontent.com/nltk/nltk_data/gh-
[nltk_data]     pages/packages/corpora/treebank.zip>:   <urlopen error
[nltk_data]     [Errno 104] Connection reset by peer>

Let's wait for the others to finish and we can retry just the failing job after

[jakirkham]

Hooray all the others passed! 🥳

Restarting the one job that had a network failure

[jakirkham]

Looks like all CI jobs passed! 🥳

[dantegd]

/merge