Add support for the project section (PEP 621)

[radoering]

Pull Request Check List

Resolves: #3332
Resolves: python-poetry/roadmap#3
Requires: python-poetry/poetry-core#708

should not be squashed

• Added tests for changed code.
• Updated documentation for changed code.

Current state:

• Full compatibility with Add support for the project section (PEP 621) poetry-core#708
• All the commands should work, especially check, lock, install, build, add, remove, init, new
• Updated documentation:
  • See preview
  • Explanation of project fields and use cases for dynamic (version, readme, classifiers) in the section "The pyproject.toml file".
  • Explanation of the difference between project.dependencies and tool.poetry.dependencies in the section "Dependency specification".
  • Updated examples to use project fields instead of tool.poetry fields.

You can install this branch with pipx as follows:

pipx install --suffix _pep621 git+https://github.com/radoering/poetry.git@pep621-support

This will install Poetry with the suffix "_pep621" so that you have to call poetry_pep621 instead of poetry. (The suffix is arbitrary, you can choose a different one.)

There is no automatic conversion of existing pyproject.toml files (yet?) but poetry check will give you information which fields should be transferred from tool.poetry to project.

[github-actions]

Deploy preview for website ready!

✅ Preview
https://website-9wjjm20ra-python-poetry.vercel.app

Built with commit fdfc7f1.
This pull request is being automatically deployed with vercel-action

[johnthagen]

After this is released, Poetry can be added to the PyPA packaging tutorial. 🚀

• https://packaging.python.org/en/latest/tutorials/packaging-projects/

[Evan-Zhao]

First of all, thank you for the great work!

I'm a bit confused by the entire extra business (which is pretty complicated itself), and especially how poetry add interacts with extras. Disclaimer: I may have wrong understanding of extras or optionals.

For this example pyproject.toml, say now I would like to add an extra feature called "dev" with dependencies black and Cython. This way pip install tvm[dev] will also install black and Cython, which a tvm developer will need (but not general users). In other words I want to do poetry add ??? black Cython, and Poetry should add such a line:

[project.optional-dependencies]
# ...
dev = ["cython (>=3.0.10,<4.0.0)", "black >= 24"]   # added

So I go to poetry add --help and find two relevant options, but it turns out neither does what I want here.
One is --optional, which specifies the dependencies are optional using a tool.poetry.dependencies specification. Makes perfect sense. The other one, --extras <extra>, is harder to follow.

Suppose I already work with tvm[highlight] (that is, with the extra highlight on) in my own environment.
If I do poetry add <dep> without -E highlight, Poetry starts to remove highlight-related deps from my virtual env, which tells me -E specifies the set of extras to consider when computing the new deps. It wasn't clear to me from the help, but this behavior is logical. But now I do poetry add -E highlight Cython, and Poetry adds this:

dependencies = [
  # ...
  "cython[highlight] (>=3.0.10,<4.0.0)"]  # added

where highlight becomes cython's extra. Is that expected? I thought if I want to add a dep with an extra I'd do poetry add "dep[extra]" instead.

It looks to me we need a command that "given existing extras e_1, e_2, ... e_n, add deps dep_1, dep_2, ..., dep_n into an optional extra e_new (if not given then add to global dependencies)",
so something like poetry add [--current-extras e]+ [--target-extra e]? [dep]+ (not actually proposing this syntax).

• In developing my project I keep multiple current-extras on, and it's kind of clumsy to always enter all of them for each add command -- is it possible at all to auto-deduce that from the lock file?
• Also I can't speak for how all of this interact with Poetry optionals and dep groups, since I'm not using these yet, but I'd love to help discuss should complexity arises there.

[radoering]

If I do poetry add without -E highlight, Poetry starts to remove highlight-related deps from my virtual env, which tells me -E specifies the set of extras to consider when computing the new deps.

That's unfortunate but not easy to change as we do not keep track of which extras (of your project) were installed. Furhter, it is not specific to this PR and thereby out of scope here. I suppose there is already an issue about this but I can't find it right now. Feel free to create a new issue. (In case it's a duplicate someone will tell you.)

If I do poetry add without -E highlight, Poetry starts to remove highlight-related deps from my virtual env, which tells me -E specifies the set of extras to consider when computing the new deps.

This conclusion is not correct. poetry add does not consider extras of your project for installing. You can probably work around this by using poetry add --lock ... to only lock (and not install in your environment) and in the end run poetry install -E highlight to install everything including your project's extra.

where highlight becomes cython's extra. Is that expected?

That's expected. See poetry add --help or the docs: "Extras to activate for the dependency."

I thought if I want to add a dep with an extra I'd do poetry add "dep[extra]" instead.

Both are equivalent.

It looks to me we need a command

--optional just adds an optional dependency but does not add it to an extra. Since that does not make sense anymore in the project section, I changed --optional to require an extra name to which the dependency is added, so now you can do: poetry add --optional highlight Cython to add Cython to the extra highlight.

In developing my project I keep multiple current-extras on, and it's kind of clumsy to always enter all of them for each add command -- is it possible at all to auto-deduce that from the lock file?

In the lockfile, only extras of dependencies are fix. All extras of your project are locked and you can/must decide which to install when running poetry install.

Also I can't speak for how all of this interact with Poetry optionals and dep groups, since I'm not using these yet

which a tvm developer will need (but not general users).

Maybe, you should consider using dependency groups instead of extras. However, dependency groups are only available during development when using Poetry and not visible in the sdist/wheel or to other tools.

All in all, there is some room for improvement considering extras but everything that goes beyond my recent change regarding --optional should probably be discussed in a separate issue.

[ferdnyc]

If I'm reading this change correctly, the syntax for adding dependencies in Poetry would change when this is committed, correct?

$ # Before, to add pytest as an optional dependency
$ poetry add --optional pytest
$ # Now, to add pytest to the "dev" extra as a dependency
$ poetry add --optional dev pytest

...Do I have that right? If so, is there a chance that users might be thrown by the syntax changing like that from one release to the next? Having to check which Poetry version you're working with to get the syntax right is kind of a bummer. To say nothing of anyone who's scripted or created aliases for Poetry commands, and has their scripts break on update because the syntax is wrong.

Would it be possible to, instead of changing --optional like that, instead provide a new syntax, and deprecate the old --optional? (It doesn't entirely make sense anymore, given that its extras support is kind of shoe-horned in.)

It's too bad poetry add already has an --extras flag, since the natural syntax for the new command would seem to be,

$ poetry add --extra dev pytest

...But given the existence of --extras as well, that seems far too confusing.

Perhaps a new subcommand would even be better?

$ # perhaps...
$ poetry extra dev --add pytest
$ # or...
$ poetry extra add dev pytest
$ poetry extra add --to dev pytest

The latter would also leave room to add additional extra subcommands, like:

• poetry extra list
• poetry extra info
• poetry extra install, perhaps even

...All of that is outside the scope of this PR, of course. I'm just a bit concerned about the semantics of a long-standing command like poetry add --optional suddenly changing "overnight".

[ferdnyc]

Another possibility, perhaps poetry add --optional without an extra specified could just add to a default extra, like dev? That way,

$ poetry add --optional pytest

could still work, and just become equivalent to poetry add --optional dev pytest. (It could even display a message warning that the old syntax is deprecated, and will be removed in the future.)

[edgarrmondragon]

...Do I have that right? If so, is there a chance that users might be thrown by the syntax changing like that from one release to the next?

fwiw I think this will be shipped in a major release, so a few breaking changes can be expected

[radoering]

Yes, PEP 621 support will only come in a major release. We still try to introduce as few breaking changes as possible but we always have to weigh up whether avoiding a breaking change requires too much maintenance effort or results in confusing UI for new users - especially in a major release where breaking changes should be expected.

Another possibility, perhaps poetry add --optional without an extra specified could just add to a default extra, like dev?

I am not convinced that this is better than failing. It might be surprising that dependencies are added to a default extra. I think sometimes it is better to fail loudly than to silently do something unexpected.

Perhaps a new subcommand would even be better?

The latter would also leave room to add additional extra subcommands, like

That might make sense but it would delay the next release even further. We can still introduce such a subcommand later and deprecate poetry add --optional to improve UI. (Of course, that does not help to avoid a breaking change now.)

All of that is outside the scope of this PR

Full ack.

[ferdnyc]

That's all fair and sensible, thanks @radoering (and @edgarrmondragon).