Feature Proposal: PEP 735 "Dependency Groups" Support

[sirosen]

What's the problem this feature will solve?

"Dependency Groups" are proposed in PEP 735 with a variety of target use cases.

In particular, the --only-deps use cases are explicitly a target for this PEP, but additional cases are listed and relevant.

Describe the solution you'd like

As the PEP author, I would like to start discussing what form, in terms of interface and implementation, pip support would take.
I would like to agree on a potential implementation path, implement it in a draft PR, and leave it pending until the PEP is accepted.

My proposal is:

• a variant of the DependencyGroupResolver I wrote moves in-tree here, along with appropriate unit tests
• a new CLI option is added to pip install, --dependency-groups, which takes a comma-delimited list of group names and attempts to resolve and install all of them from pyproject.toml in the current working directory

For example,

# in pyproject.toml
[dependency-groups]
test = ["pytest"]
typing = ["pyright"]

# on the CLI
pip install --dependency-groups test,typing

# equivalent to
pip install pytest pyright

Would the pip maintainers be amenable to this addition? Is there an alternative, similar strategy which I could implement?

Alternative Solutions

If pip does not gain support for Dependency Groups, it is possible to workaround the gap.
Primarily, external tooling like dependency-groups can be used to resolve groups and pass them to pip.

For example, in a *nix environment, the following usage covers most cases:

groups=$(python -m dependency_groups pytest typing | tr '\n' ' ')
pip install $groups

However, such usages will always be workarounds. pip support provides a superior developer experience.

Additional context

PEP 735 is in Draft, and will be submitted soon. At time of writing, it is not final or accepted, but no further changes to the contents are planned.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

Some questions around the UI.

How would something like pip install requests --dependency-groups xxx work? Is the flag intended to only be valid for installing from source? What about pip install requests --no-binary requests --dependency-groups xxx in that case?

How will the flag work with multiple requirements? If I do pip install --dependency-groups foo ./a ./b ./c is the dependency group looked up for all of a, b, and c? Or is this invalid usage?

Basically, I don't think we have an existing flag that works like I'd expect this one to, which might be a usability problem.

[RonnyPfannschmidt]

I'd consider pip the wrong place for managing this in a normal way

IMHO it's different enough from normal install that it warrants a distinct command to install dependency groups of a specific source folder

Tools like hatch, rye and poetry should be the place of closer integration

[sirosen]

I am a bit concerned about meeting the target use cases for the PEP without pip support. It would be possible, but the user experience is worse in several cases.

IMHO it's different enough from normal install that it warrants a distinct command to install dependency groups of a specific source folder

That may be, but I see the feature as similar to -r. The main difference being that the path lookup is implicit in the UX I proposed -- it needn't be. We can discuss usages in which the path to pyproject.toml is given explicitly. (More on this below)

Tools like hatch, rye and poetry should be the place of closer integration

I think the answer here is "and" not "or". Dependency Groups will have the greatest positive impact if they are supported by installers, environment managers, and build backends.

I've started with pip and setuptools. Perhaps this is a strategy mistake and I should begin from younger tools like flit.

---

Before I dive into Paul's specific scenarios, I want to start from the high level concern, make sure I understand it, and consider whether or not the UX I proposed is correct.

Basically, I don't think we have an existing flag that works like I'd expect this one to, which might be a usability problem.

Is the issue that this is expected to expand to a set of requirements, whereas -r expands to a set of arguments? Or is it the implicit lookup of ./pyproject.toml? Both of these? Something else?

I'm trying to understand if the issue would be mitigated by making the toml file explicit, e.g., pip install --dependency-group pyproject.toml:foo --dependency-group ../otherproj/pyproject.toml:bar.

[RonnyPfannschmidt]

I believe the proposed installation ux clearly shows my 3rd point

[pfmoore]

I think the problem for me was that it wasn't clear that you're proposing that --dependency-groups works the same way as --requirement. In particular, in the pip install help, we currently have

python -m pip install [options] <requirement specifier> [package-index-options] ...
python -m pip install [options] -r <requirements file> [package-index-options] ...
python -m pip install [options] [-e] <vcs project url> ...
python -m pip install [options] [-e] <local project path> ...
python -m pip install [options] <archive url/path> ...

If I understand, you're proposing to add a new form here:

python -m pip install [options] --dependency-groups <list of groups>  [package-index-options] ...

This would look in a pyproject.toml in the current directory and extract the list of requirements to install from the specified dependency groups in there. Looking at it that way seems a lot more plausible, and eliminates all of my objections above. We do allow -r and "normal" requirements to be combined, even though the syntax above doesn't show it. I'd assume --dependency-groups would work the same.

Basically, I don't think we have an existing flag that works like I'd expect this one to, which might be a usability problem.

That was incorrect, we do, it's -r. If we mirror the behaviour of -r, that should be OK. Sorry for the confusion here.

There's some background information available in #8049. The conclusion there was that we rejected the feature request, but dependency groups are much more specific, and will be standards-based, so I think it's within scope for pip to support them.

Also, there may be some edge cases that won't work, because requirements files are much richer than dependency groups. For example, --dependency-groups won't work well with hash checking mode, as (unlike requirement files) dependency groups don't offer a way to specify a hash. I think that's fine, but it might be worth explicitly documenting any limitations we are aware of somewhere, so we don't get user reports saying "hash checking fails with dependency groups" and expecting us to fix this.

[pfmoore]

We can discuss usages in which the path to pyproject.toml is given explicitly.

Strong -1 on me for allowing anything but "read dependency groups from pyproject.toml in the current directory". Finding the pyproject.toml for some idea of the "current project" should be left to project management tools, and explicitly stating the path to pyproject.toml is unnecessary complexity that (as far as I'm aware) there's no valid use case for.

[notatallshaw]

Strong -1 on me for allowing anything but "read dependency groups from pyproject.toml in the current directory". Finding the pyproject.toml for some idea of the "current project" should be left to project management tools, and explicitly stating the path to pyproject.toml is unnecessary complexity that (as far as I'm aware) there's no valid use case for.

So one of the repos I work with is a messy monorepo, in that repo we would never put a pyproject.toml in there because it might stomp all over the settings of one team or another.

This has never been a problem because front end and utility tools, like pip, pre-commit, linters, formatters, etc. allow you to specify where you want the configuration / requirements to come from. And backend tools that do not allow this are generally not suitable for the repo anyway i.e. we never want to build a package in the root of the repo.

Were pip to use this as a "requirements-like install file but with structured options" it would be a departure from pip's current approach to allow the user to specify the path of the requirements file or constraints file, and other front end tools in general.

If you don't consider "messy monorepo" as a valid use case, fair enough, but we're not trying to build a package from it, so it would be annoying for frontend tools, that are not project management tools, to say how we should structure our project.

That said, this isn’t a hill I want to die on, just want to give an alternative view point.