Use the [project] section in pyproject.toml according to PEP-621

[MartinThoma]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

PEP621 (Storing project metadata in pyproject.toml) and PEP631 (Dependency specification in pyproject.toml based on PEP 508) define a clear way how metadata and dependencies should be stored. Both of them have the status "accepted". It would be good if poetry would use that.

For example, I've noticed that poetry init generates a pretty similar pyproject.toml, but puts the metadata in a poetry-specific section.

[sinoroc]

The status of PEP 621 is still "Draft". But yes, I guess it's a valid request (assuming it actually gets accepted and depending under which form). It might take a while though... the way I see it, there are still too many uncertainties

[MartinThoma]

Uh, interesting. My bad. However, PEP-631 is accepted. Do you see uncertainties there as well?

[sinoroc]

At some point, I thought there was an intention to merge the text of 631 into 621, to avoid having to refer to two documents, since 631 is rather short and unsurprising. Don't know if it will happen, or if it stays as 2 separate documents. I lost track of that conversation. And there are some more points of 621 that are still being discussed.

Now, if I put myself in the shoes of poetry:

• PEP 621 might seem like a step to the side, no big changes forward or backward, but we move from non standard to standard. So probably a slight positive.
• PEP 631 could definitely be seen as a step backwards.

Personally I am curious to see how poetry will handle this. There is always the dynamic escape hatch for dependencies. If I were poetry, I would probably try to offer both dependency notations (the existing poetry notation as dynamic and the new/old PEP 631 one). I had suggested a compromise hybrid notation here but it didn't gain any traction at all (probably because too complex to parse, I don't know).

[MartinThoma]

there was an intention to merge the text of 631 into 621

It's mentioned in the abstract of PEP-0631. However, I don't know when this will happen and where the discussion around that is.

PEP 631 could definitely be seen as a step backwards.

I'm surprised by that. Why do you think so? To me, it seems like a step towards more standardization and thus positive.

I would probably try to offer both dependency notations

Yes, I think that would be desirable just to not break a working setup for anybody. Is the poetry dependency notation more powerful or even different at all?

[sinoroc]

Is the poetry dependency notation more powerful or even different at all?

It's all in the discussions, particularly this one, I don't want to repeat things and take them out of context:

• https://discuss.python.org/t/pep-621-how-to-specify-dependencies/4599

[finswimmer]

The discussion about how the dependencies should be defined within PEP-621 was quite said and frustrating for us as poetry maintainers. But I will not go into details now...

As @sdispater has said somewhere in the discussion, poetry will probably not be one of the earlier adopters of PEP-621 now PEP-631 is accepted. We have a problem here now: On the one hand, we want to support as many standards as possible and on the other, user experience is an important topic for us. And we believe that PEP-631 will be a step backwards compared to what we have now. A dilemma we have to discuss. :(

[stephen-dexda]

PEP 621's status is now provisionally accepted as of 8 days ago.

From the point of view of a package dev., having a build tool-independent metadata specification in pyproject.toml is useful to be able to parse metadata from the file without adding build tool-specific references elsewhere, and without duplicating information.

Use case examples:

• project.scripts.*: when building container images, to determine which file to reference as to be executed.
• project.requires-python: to determine which Python versions to run tests against.
• project.optional-dependencies.test: to install test dependencies.

I have been using/parsing all of the above based on the PEP 621/631 drafts, and duplicating this info in pyproject.toml for now from the tool.poetry sections.

Fundamentally, project metadata is not only used by build tools, so having Poetry support (provisionally) accepted standards for these allows the developer to use the metadata for other cases without duplication, and having this support from Poetry would be fantastic :)

[stephen-dexda]

Additionally, PEP 621 is now referenced at https://packaging.python.org/specifications/declaring-project-metadata/.

[stephen-dexda]

As part of this, poetry version <rule> should bump project.version as well as/instead of tool.poetry.version, depending which is/are present.

[yoursvivek]

Status of PEP 621 is now: Final.

• Announcment by Bret Cannon on python discussion forum.
• PR on python/peps is now merged in master.

[mvaled]

The way I see it, poetry can start by supporting PEP 621. PEP 631 (actually PEP 508) requires some changes in the way versions are dealt with in Poetry (e.g ~= 3.6 instead of ^3.6).

[stephen-dexda]

PEP 621 was updated to incorporate the decision on PEP 631 vs. PEP 633; it requires PEP 508 version strings.

[zubieta]

Hi poetry maintainers, do you think is there a way to start supporting everything else in pep 621 while the dependency list discussion happens? I don't want to misrepresent anyone so correct me if I'm wrong, as I understood the discussion it looks like pretty much everything else (name, version, python_requires, etc) was generally accepted and shouldn't impact poetry usability.

PS. Thank you for this awesome piece of software.

[eli-schwartz]

If even permitting users to specify PEP compliant dependency versions is a dealbreaker (remember, people wanting poetry style versions can just specify that dependencies is a dynamic metadata key), perhaps poetry could refuse to support any pyproject.toml that doesn't specify it in dynamic?

"Error: poetry requires specifying dynamic dependencies in order to manage it differently".

Then people could use the PEP, have a PEP compliant pyproject.toml, and get some benefits of the PEP (and test out using it in poetry), even if poetry doesn't accept every possible way of following the PEP.

[sinoroc]

Last time I checked (which was a long time ago already, I must admit), I also came to a similar conclusion, poetry could relatively easily move to PEP621 and use the dynamic escape hatch for dependencies.

---

To the people in this thread asking for support of PEP621, if I may, honest question: What is the motivation for you to see PEP621 adopted by poetry? Is there any advantage already today to have PEP621? What would be the benefits for your project today? (Medium- long-term, I see the advantages, obviously)

[stephen-dexda]

The solution seems pretty straightforward to me in principle - fully/properly support PEP621 (and that includes dependency specification per PEP631), and if there is some form of dependency specification that can't be expressed in PEP631 format, then either that can be expressed in a dynamic section, or Poetry can still use its own sections (or some combination of both, e.g. use-poetry-section-for-deps = true in dynamic).

PEP621 doesn't have to support everything Poetry's config. supports for it to be supported by Poetry, it doesn't make sense to look at this issue with that assumption. It just needs to be able to sufficiently define project metadata in at least some cases for Poetry to be able to use [project] to build the project. Which it does (and probably does in most cases).

sinoroc, see #3332 (comment) in answer to your last question.

[stephen-dexda]

If even permitting users to specify PEP compliant dependency versions is a dealbreaker (remember, people wanting poetry style versions can just specify that dependencies is a dynamic metadata key), perhaps poetry could refuse to support any pyproject.toml that doesn't specify it in dynamic?

"Error: poetry requires specifying dynamic dependencies in order to manage it differently".

Honestly, this seems like the worst of all worlds.

[eli-schwartz]

Honestly, this seems like the worst of all worlds.

I'm not irrationally suggesting that poetry becomes worse as a result of the PEP because of the disagreement over version descriptions. Using dynamic is a perfectly reasonable escape hatch for people who prefer poetry's format.

Given some people do seem to think that merely offering the option to use PEP 631 is "bad", I figured a partial implementation is better than no implementation, because a partial implementation is partially useful whereas no implementation has no usefulness.

I don't actually think this is a remotely ideal situation...

[zubieta]

Last time I checked (which was a long time ago already, I must admit), I also came to a similar conclusion, poetry could relatively easily move to PEP621 and use the dynamic escape hatch for dependencies.

To the people in this thread asking for support of PEP621, if I may, honest question: What is the motivation for you to see PEP621 adopted by poetry? Is there any advantage already today to have PEP621? What would be the benefits for your project today? (Medium- long-term, I see the advantages, obviously)

@sinoroc I'm personally more interested in having the non-dependencies metadata in a single place so that several tools/scripts can reuse it without the hassle of keep it in sync. For example for a couple of projects I use poetry and towncrier:

[tool.poetry]
name = "mypackage"
version = "0.1.0"
description = "My awesome package."

[tool.towncrier]
package = "mypackage"
package_dir = "src"
filename = "NEWS.rst"
version = "0.1.0"

Like this example I have more related to documentation and publishing tools, albeit mostly for custom scripts atm.

[RootLUG]

Last time I checked (which was a long time ago already, I must admit), I also came to a similar conclusion, poetry could relatively easily move to PEP621 and use the dynamic escape hatch for dependencies.

To the people in this thread asking for support of PEP621, if I may, honest question: What is the motivation for you to see PEP621 adopted by poetry? Is there any advantage already today to have PEP621? What would be the benefits for your project today? (Medium- long-term, I see the advantages, obviously)

One of the big advantages of using PEP621 would also be that 3rd party tools would understand the dependencies. I for example have a project that is doing static analysis of a code and also dependencies (flagging outdated ones, potentially malicious, vulnerable packages, etc). If every tool has its own format of how dependencies are declared then the static analyzer must have a separate support for each of these tools which are quite hard to maintain over time.

If all the tools start migrating to use the PEP621 then that would mean also a more secure python ecosystem as the static analyzers can all use the same metadata information regardless if the project is using poetry, setuptools, flit, or anything else.

There are several other metadata info in the toml file that is very helpful for static analyzers and auditing tools which is very painful to support and maintain given all the different formats that each packaging tool has.

[finswimmer]

PEP621 in general is good idea and can even make poetry's live easier. @sdispater is one of the authors of this PEP.

The hard question is how the dependency specification in poetry project should look like. poetry projects doesn't have just run time dependencies, they also have dev-dependencies and more dependency groups are planned. How do we want to be able to use poetry specific flags like develop=True or allow-prereleases = true. As poetry evolves more flags will arise.

[stephen-dexda]

@finswimmer I answered that exact question in #3332 (comment).

[finswimmer]

For sure dynamic is an option. But - at least for poetry - the dependencies are the most important metadata and it would be a bit strange if poetry tries to push other packaging tool to write there dependency according to PEP631 and doesn't do this itself.

[sinoroc]

poetry projects doesn't have just run time dependencies, they also have dev-dependencies

On this one specific thing, I wish packaging tools would agree on some conventions here. I for example like and use the convention of having a dev extra for those development dependencies. Works amazingly well with setuptools and tox for example.

[gbdlin]

Is there any summary what we will lose when moving project dependencies into the PEP-621/PEP-631 specification?

I know optional dependencies/extras will be defined quite different and wouldn't allow to automatically re-use the same packet version in multiple extras groups in the same way, but there is another option for that and should already work in other build backends. Basically, you can define your dependencies like:

[project.optional-dependencies]
test = [
  "pytest < 5.0.0",
  "pytest-cov[all]"
]
lint = [
  "black",
  "flake8"
]
ci = [
  "beaglevote[test]",
  "beaglevote[lint]"
]

And the ci extras should include everything from the test and lint. We can support that syntax officially and document it as something recommended.

Another thing I know of is defining the source of specific packages - I understand why [project] is not supporting that, as this is not something for the packages, but for non-package projects it is at least useful.

[johnthagen]

An example of a project migrating from Poetry to PDM in part of the missing PEP 621 support

• Support PDM openapi-generators/openapi-python-client#920

[tigerhawkvok]

I think Poetry's syntax is better and more powerful, but it's important to support PEP621.

I propose that:

• [project] may always exist
• requires-python may exist in either tool.poetry.dependencies or project, but only in one.
• If tool.poetry.dependencies exists, neither dependencies or optional-dependencies are allowed as keys in project. Having both raises an error.
• If tool.poetry.dependencies does not exist, dependencies will be resolved as per PEP621
• Poetry adds a function (maybe something like poetry pep) to resolve a tool.poetry.dependencies to PEP621 dependencies and delete the corresponding section in pyproject.toml, in case anyone wants to lock it down that way. (I would personally never use such a function, but it removes the friction of "well I don't want to get locked into the poetry build structure so I will just never use poetry"). Alternatively a poetry migrate, toggling the dependencies back and forth.

The upshot of my suggestion is to not try to support some complicated blend. Let people pick one or the other, and raise an error if it's not just one or the other. Should be much easier to develop.

[eli-schwartz]

The upshot of my suggestion is to not try to support some complicated blend. Let people pick one or the other, and raise an error if it's not just one or the other. Should be much easier to develop.

I think I can summarize your fairly lengthy description.

• poetry is updated to support pep 621
  • pep 621 states that metadata in pep 621 is canonical and it is a fatal error to define a tool specific version
  • pep 621 states that metadata can be declared as dynamic, in which case it MUST NOT be defined via pep 621, but is defined via (existing) tool specific sections

This has already been suggested but I believe poetry didn't want to do that.

[abravalheri]

[project] may always exist\nrequires-python may exist in either tool.poetry.dependencies or project, but only in one.\nIf tool.poetry.dependencies exists, neither dependencies or optional-dependencies are allowed as keys in project. Having both raises an error.

I don't know if I understood this proposal correctly, but if the idea is allow [project] and tool.poetry.dependencies to be defined simultaneously, then according to the PEP, project.dynamic must be validated in such scenarios.

It should:

1. Be necessarily defined by the user.
2. Contain "dependencies".

[Turakar]

Have we already discussed the following option: Whenever Poetry performs dependency resolution, i.e., when it makes changes to poetry.lock, it could also write a PEP 621 compliant set of dependencies to [project], effectively overwriting the contents there with a big comment warning # These dependencies are automatically generated by Poetry based on the contents of [tool.poetry.dependencies]..

If all other Metadata is equally well specifiable in [project], we might want to deprecate writing it to Poetry-specific tables, otherwise we could rely on overwriting here as well. Most likely, a feature flag would be desirable to turn this behavior on or off.

This proposal becomes difficult if there are Poetry dependency expressions which are inexpressible in PEP 621. I do not really know a solution to this if these cases have actual usefulness and are not just weird corner cases.

[merwok]

pyproject is not meant to be a lock file – dependencies in pyproject are not a full list of pinned dependencies.

[flying-sheep]

I think Poetry's syntax is better and more powerful

That’s subjective, and false, respectively.

Regarding “better”, I think it’s worse since ^ encourages specifying upper bounds, which you shouldn’t do in almost all cases.

Regarding “more powerful”, see above the distinction between abstract and concrete dependencies.

• project.dependencies is for what you need. E.g. if you want to use a feature from PyTorch 2.1, you put torch >=2.1 in there.
• If you want to specify how to resolve dependencies, project.dependencies is not the place for that. E.g. if something like PyTorch needs additional messing to install properly, you put that somewhere else (e.g. in a environment.yaml or a pixi.toml or a nix config file or maybe in a tool.poetry.resolutions table or something).

[NeilGirdhar]

• If you want to specify how to resolve dependencies, project.dependencies is not the place for that.

You may want to watch this proposal (or its descendants when it grows them) so that you can participate in it.

[flying-sheep]

Thanks. I personally don’t need this that much. I think people who do should contribute there, so they can get all the Poetry resolution niceness in a standardized way, everywhere.

[eli-schwartz]

Regarding “more powerful”, see above the distinction between abstract and concrete dependencies.

Huh, and I thought you were going to say something about how poetry's dependency syntax cannot be more powerful than the PEP 621 syntax since the poetry syntax is, quite literally, compiled by poetry into the PEP 621 syntax (PEP 508 strings) in the process of being written into the wheel metadata. This was, after all, a major factor in poetry's PEP 633 being rejected.

[flying-sheep]

You’re right. The metadata part (abstract dependencies) is just as powerful. Poetry mixes that with package sources and path dependencies, which are resolution features (concrete dependencies), and therefore shouldn’t be part of the dependency array.

Of course, package sources and path dependencies are super useful, but extricating them from the dependencies array doesn’t make anything less powerful, just differently structured.

[radoering]

#9135 should be in a testable state now. (Consider it an alpha version.) Please try it out and report issues you encounter in the PR.