Distribution of py3-law in cmssw

[smuzaffar]

Hi,
While building CMSSW software stack for python 3.12 we noticed that py3-law ( which was added to our software stack via cms-sw/cmsdist#6649 ) did not build due to its dependencies python-daemon and luigi failed to build for python 3.12 ( along with updated setuptools).

As nothing in cmssw directly use law or any of its dependencies, do we need to include this in our distribution? @cms-sw/analysis-l2, if no objections then we would like to drop law from cmssw software stack and one can use python virtual environment in cmssw to locally install it once it supports python 3.12?

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @smuzaffar.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

assign analysis

(even if analysis is practically without real maintenance effort)

[cmsbuild]

New categories assigned: analysis

@tvami you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

Tagging @riga who authored cms-sw/cmsdist#6649, and @clelange @lenzip @anigamova for CAT.

[clelange]

Thanks for the ping -- luigi seems to support Python 3.12 though, it's stated on their PyPI page?
@smuzaffar do you have any more details on what exactly fails to build?
@riga can probably comment better how inclusion in CMSSW helps with law workflows/whether this is still needed.

[iarspider]

@clelange the issue is with python-daemon (which is required for luigi, which in turn is required for law). This is the error reported:

Processing ./src.tar.gz
  Preparing metadata (pyproject.toml): started
  Running command Preparing metadata (pyproject.toml)
  Traceback (most recent call last):
    File "/build/razumov/cmsbuild/BUILD/el8_amd64_gcc12/external/py3-pip/24.0-b1aa17d03e426f17ac26e10ec6f018b6/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/build/razumov/cmsbuild/BUILD/el8_amd64_gcc12/external/py3-pip/24.0-b1aa17d03e426f17ac26e10ec6f018b6/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/build/razumov/cmsbuild/BUILD/el8_amd64_gcc12/external/py3-pip/24.0-b1aa17d03e426f17ac26e10ec6f018b6/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
      return hook(metadata_directory, config_settings)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/build/razumov/cmsbuild/BUILD/el8_amd64_gcc12/external/py3-setuptools/72.1.0-f09b74ebddf75c63bf10f1dadf250d18/lib/python3.12/site-packages/setuptools/build_meta.py", line 368, in prepare_metadata_for_build_wheel
      self.run_setup()
    File "/build/razumov/cmsbuild/BUILD/el8_amd64_gcc12/external/py3-setuptools/72.1.0-f09b74ebddf75c63bf10f1dadf250d18/lib/python3.12/site-packages/setuptools/build_meta.py", line 313, in run_setup
      exec(code, locals())
    File "<string>", line 23, in <module>
    File "/build/razumov/cmsbuild/BUILD/BUILD/el8_amd64_gcc12/external/py3-python-daemon/3.0.1-73c71b7d6919283fd1c0d9f34dc746ed/cmsdist-tmp/pip-req-build-vmyvfrut/version.py", line 39, in <module>
      import setuptools.extern.packaging.version
  ModuleNotFoundError: No module named 'setuptools.extern'
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

[clelange]

OK, thanks @iarspider -- maybe this is something @riga can fix (if it's still advantageous to have law in CMSSW).

[riga]

Hi! This looks like an inconsistency in the dependency tree within cmsdist. I guess updating setuptools to 72.2.0 should fix the issue.

As far as I know, 3 projects in 2 POGs use the version of law that is shipped with the cmssw software stack. In case the above suggestion helps, we could keep it for now. If not, we should reiterate.

[smuzaffar]

@iarspider , can you please try to update setuptools to 72.2.0 ?

[iarspider]

@iarspider , can you please try to update setuptools to 72.2.0 ?

That doesn't help:

+ pip3 list --disable-pip-version-check
Package    Version
---------- -------------------
docutils   0.21.2
flit_core  3.9.0
lockfile   0.12.2
pip        24.0
setuptools 72.2.0.post20240816
wheel      0.43.0
+ pip3 install --no-clean --no-deps --no-index --no-build-isolation --no-cache-dir --disable-pip-version-check --user -v src.tar.gz
Using pip 24.0 from /build/razumov/cmsbuild/BUILD/el8_amd64_gcc12/external/py3-pip/24.0-c9f46e7f7a7c6e370ff8885a7adaf71d/lib/python3.12/site-packages/pip (python 3.12)
Processing ./src.tar.gz
  Preparing metadata (pyproject.toml): started
  Running command Preparing metadata (pyproject.toml)
  Traceback (most recent call last):
    File "/build/razumov/cmsbuild/BUILD/el8_amd64_gcc12/external/py3-pip/24.0-c9f46e7f7a7c6e370ff8885a7adaf71d/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/build/razumov/cmsbuild/BUILD/el8_amd64_gcc12/external/py3-pip/24.0-c9f46e7f7a7c6e370ff8885a7adaf71d/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/build/razumov/cmsbuild/BUILD/el8_amd64_gcc12/external/py3-pip/24.0-c9f46e7f7a7c6e370ff8885a7adaf71d/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
      return hook(metadata_directory, config_settings)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/build/razumov/cmsbuild/BUILD/el8_amd64_gcc12/external/py3-setuptools/72.2.0-40b075488a7411ecb8baa658d874a69e/lib/python3.12/site-packages/setuptools/build_meta.py", line 368, in prepare_metadata_for_build_wheel
      self.run_setup()
    File "/build/razumov/cmsbuild/BUILD/el8_amd64_gcc12/external/py3-setuptools/72.2.0-40b075488a7411ecb8baa658d874a69e/lib/python3.12/site-packages/setuptools/build_meta.py", line 313, in run_setup
      exec(code, locals())
    File "<string>", line 23, in <module>
    File "/build/razumov/cmsbuild/BUILD/BUILD/el8_amd64_gcc12/external/py3-python-daemon/3.0.1-ff3d9f9e9bf619cab0c2246d5c6b56d6/cmsdist-tmp/pip-req-build-l4n9rxaf/version.py", line 39, in <module>
      import setuptools.extern.packaging.version
  ModuleNotFoundError: No module named 'setuptools.extern'
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /build/razumov/cmsbuild/BUILD/el8_amd64_gcc12/external/python3/3.12.4-8b38497c8dc5715caa8d5d9c0b045e13/bin/python3 /build/razumov/cmsbuild/BUILD/el8_amd64_gcc12/external/py3-pip/24.0-c9f46e7f7a7c6e370ff8885a7adaf71d/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py prepare_metadata_for_build_wheel /build/razumov/cmsbuild/BUILD/BUILD/el8_amd64_gcc12/external/py3-python-daemon/3.0.1-ff3d9f9e9bf619cab0c2246d5c6b56d6/cmsdist-tmp/tmprew706v1
  cwd: /build/razumov/cmsbuild/BUILD/BUILD/el8_amd64_gcc12/external/py3-python-daemon/3.0.1-ff3d9f9e9bf619cab0c2246d5c6b56d6/cmsdist-tmp/pip-req-build-l4n9rxaf
  Preparing metadata (pyproject.toml): finished with status 'error'
error: metadata-generation-failed

@riga setuptools.extern is an undocumented internal module of setuptools, it shouldn't be used link.

But I think I have found how to fix python-daemon for now. We can close this issue.