PEP 621 support for dynamic metadata

[Alex-ley-scrub]

In PEP 621 is the dynamic stuff, is there any plan to support it for dependencies and optional-dependencies?

• How is this important to you? How would you use it?
  • we have built a backend that hosts NLP models and returns predictions
  • until now this was 100% python (flask, pytorch, transformers, pandas, etc.)
  • I started writing some rust for some bottleneck code and calling it from python
  • I used pyo3 and maturin to build it and test it locally - all works well, fantastic tools 👏
  • I need to migrate our current build system away from setuptools to maturin to distribute my application with the new rust compiled library to the team. Right now we use conda and pip and setuptools defined in pyproject.toml and we all run commands such as git pull and conda env update --file environment.dev.yml --prune and pip install -e .[dev,test] to ensure we are all developing on the latest code in the latest environment with the latest dependencies. We pin all versions of our primary packages in 3 requirements files requirements.txt (prod), requirements.dev.txt (dev), and requirements.test.txt (test) so that we can separate out the requirements. Right now setuptools handles this well for a pure python program, such that pip install -e . (prod/base requirements) or pip install -e[dev,test] (prod + dev + test requirements) both work really well.
• Can you think of any alternatives?
  • setuptools or maybe https://github.com/PyO3/setuptools-rust etc.
• Do you have any ideas about how it can be implemented? Are you willing/able to implement it? Do you need mentoring?
  • with some guidance I may be able to implement it but that depends a lot on the existing project/repo structure etc. (I haven't looked yet)

some examples/links:

https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html#dynamic-metadata
https://peps.python.org/pep-0621/#dynamic

pyproject.toml:

[project]
name = "my_package"
dynamic = ["version", "readme", "dependencies", "optional-dependencies"]

[tool.maturin.dynamic]
version = {attr = "my_package.__version__"}
readme = {file = "README.md"}
dependencies = {file = "requirements.txt"}

[tool.maturin.dynamic.optional-dependencies]
dev = {file = "requirements.dev.txt"}
test = {file = "requirements.test.txt"}

There is something very similar that setuptools supports in setup.cfg files too (https://setuptools.pypa.io/en/latest/userguide/declarative_config.html#options):

[metadata]
name = "my_package"
version = attr: my_package.__version__
long_description = file: README.md
long_description_content_type = text/markdown

[options]
packages = find:
zip_safe = False
python_requires = >=3.9
include_package_data = True
install_requires = file: requirements.txt

[options.extras_require]
dev = file: requirements.dev.txt
test = file: requirements.test.txt

the analogous idea in the older setup.py was (and the code to implement this in maturin may not need to be much more complicated than this I guess - probably some filtering of comment/pip flag lines etc.):

import os
from setuptools import find_packages, setup

dirname = os.path.abspath(os.path.dirname(__file__))

with open(os.path.join(dirname, "README.md"), "r", encoding="utf-8") as f:
    long_description = f.read()

with open(os.path.join(dirname, "requirements.txt")) as f:
    required = f.read().splitlines()

with open(os.path.join(dirname, "requirements.test.txt")) as f:
    test_required = f.read().splitlines()

with open(os.path.join(dirname, "requirements.dev.txt")) as f:
    dev_required = f.read().splitlines()
    
setup(
    name="my_package",  # Replace with your own username
    version=__version__,
    author=__author__,
    author_email=__author_email__,
    license=__license__,
    description="my_package for xyz",
    long_description=long_description,
    long_description_content_type="text/markdown",
    package_dir={"my_package": "my_package"},
    packages=find_packages(where=".", exclude=["*.tests"]),
    python_requires=">=3.9", 
    install_requires=required,
    # https://setuptools.readthedocs.io/en/latest/userguide/dependency_management.html#optional-dependencies:
    extras_require={"dev": dev_required, "test": test_required},
    zip_safe=False,
)

[messense]

• Parsing code can be added here:

  maturin/src/pyproject_toml.rs

  Lines 110 to 111 in 765915c

  	pub struct ToolMaturin {
  	// maturin specific options

• We'd need to adjust wheel metadata here:

  maturin/src/metadata.rs

  Lines 85 to 92 in 765915c

  	/// Merge metadata with pyproject.toml, where pyproject.toml takes precedence
  	///
  	/// pyproject_dir must be the directory containing pyproject.toml
  	pub fn merge_pyproject_toml(
  	&mut self,
  	pyproject_dir: impl AsRef<Path>,
  	pyproject_toml: &PyProjectToml,
  	) -> Result<()> {

[messense]

Implementing attr support might be trick, ideally we don't want to depend on a working Python interpreter to such trivial task (ast.literal_eval), but pulling in heavy dependencies like rustpython-parser increases compile time by a lot thus isn't worth it.

I guess if it's a must have feature, we have to require a working Python interpreter for simplicity.

[messense]

See also scikit-build/scikit-build-core#230

[Alex-ley-scrub]

Thanks @messense. I am a lot better at python than I am at Rust currently. I am learning Rust to speedup our bottlenecks as described above. So I am not (yet) very proficient in it. That said, it is a beautiful language and my first impressions are very good. It will take me a little bit of time to read through the codebase and to understand what parts I would need to change. Thanks a lot for linking the relevant sections. Is it worth waiting a little while before we implement if the PEP is still being finalized? Or are you happy with a BETA implementation like setuptools has? I do like the suggested approach in scikit-build/scikit-build-core#230 - that is much simpler than the backend specific [tool.maturin.dynamic] and [tool.setuptools.dynamic]

# After
[project.dynamic]
version = {attr="mymod.__version__"}
dependencies = {file="requeriments.in"}
optional-dependencies.dev = {file="dev-requeriments.in"}
optional-dependencies.test = {file="test-requeriments.in"}

I would also personally be happy to omit attr support for now. I see much less value in this than in the file option.

[konstin]

I'm not sure if that can cover your use case, but two notes from my side:

version = {attr="mymod.__version__"}: I'd recommend putting the version into pyproject.toml and reading it with importlib.metadata

For the requirements, pip-tools supports PEP 621. Would keeping all dependencies in pyproject.toml and exporting them with pip-compile (or any other tool of your choice) work for you? I know it's suboptimal to have this duplication, but i expect that eventually all relevant tools will be able to install from pyproject.toml so it's only a temporary solution.

In general the big disadvantages of the PEP proposal are that we need to install the requirements beforehand (while currently we can just build an abi3 rust crate and put in an archive, no python interpreter involved), it's performance overhead with extra error surface (launching the provider in a new python interpreter) and static analysis (including someone looking at pyproject.toml) can't see the metadata anymore. That's why i try hard to find solutions that work without the dynamic provider syntax and only use it if there's no other way.

[wiktor-k]

version = {attr="mymod.version"}: I'd recommend putting the version into pyproject.toml and reading it with importlib.metadata

One use-case not covered with statically putting it in pyproject.toml is automatically deriving version from the output of git describe. Something like this:

__version__ = subprocess.check_output(['git', 'describe']).decode('utf8').lstrip('v').strip()

(I've found this ticket while trying to simplify version field in my project).

[jamestwebber]

I would like to see this as well, it matches my current workflow for writing python packages and is particularly nice when configuring optional dependencies.

A maturin-specific feature that might be nice is to pull the package version from Cargo.toml. There are probably situations where one would want to version the python package and rust crate separately, but I suspect most of the time people will want to keep them in lockstep.

[nils-werner]

Maybe I'm misreading what you're trying to do, but I would recommend to maybe overthink your deployment strategy a little bit, and possibly change it to something that cargo and poetry are doing. In a gist:

Your library should have flexible dependencies. That means in your setup.py, pyproject.toml or Cargo.toml, there should be something akin to

[dependencies]
pytest = "^7.0.0"

These flexibly declare what version you need to have for this library to run.

Your deployments on the other side should have locked dependencies, to ensure reproducible builds and rollbacks. That means your poetry.lock or Cargo.lock should have the actual concrete version number and possibly a hash value, like

[[package]]
name = "pytest"
version = "7.4.2"
description = "pytest: simple powerful testing with Python"
optional = false
python-versions = ">=3.7"
files = [
    {file = "pytest-7.4.2-py3-none-any.whl", hash = "sha256:1d881c6124e08ff0a1bb75ba3ec0bfd8b5354a01c194ddd5a0a870a48d99b002"},
    {file = "pytest-7.4.2.tar.gz", hash = "sha256:a766259cfab564a2ad52cb1aae1b881a75c3eb7e34ca3779697c23ed47c47069"},
]

That way you ensure that your library remains more flexible and open for reuse in other projects with other deployments, and your deployments are reproducible using lockfiles.

As soon as you start putting the content of requirements.txt or environment.yml files in your library dependencies, you're nailing the library to the deployment and are making reuse almost entirely impossible.

Now, of course your library still can contain a lockfile or requirements.txt (Cargo.lock exists in every single crate, after all). But this file is only meant for working on the library, not for using the library. Take cargo as an example: If you install a few crates, cargo will completely ignore the Cargo.lock files from all those crates, and will only consider the one Cargo.lock of the project that you're working on.

[alonbl]

Hello,

We also require setting dynamic version for CI to be able to set the version components.
The version = {file = "version.txt"} is sufficient, no need to call back into python.
Another option is to have a set of substitution variables in maturin configuration so these can be specified in pyproject.toml.
Best is to be able to pass these as build options using python -m build --config-setting KEY=VALUE.

Do you know any other method of passing CI settings into the build process without overwriting git managed files?

Regards,