Unhandled error from already-installed dependency with non-standard dependency specifier

[edmorley]

Description

In #12063 support for non-standard (ie: broken) dependency specifiers was first deprecated, and then in pip 24.1.0, removed entirely.

As such pip now ignores such broken packages when resolving dependencies during package installation - meaning they can no longer be installed. This is fine, and currently working as expected.

However, what it not working as expected, is that if there are any already-installed dependencies with those dependency specifiers then pip install fails with an internal error and does not explain what happened or how to resolve it.

Expected behavior

Using the steps to reproduce below, pip should either:

1. Succeed as a no-op, since the requested dependency is satisfied (ie: there is nothing to install/uninstall), since "celery" is satisfied by any installed version. (And in fact the log output below even says "Requirement already satisfied".)
2. Or, successfully uninstall the broken package and install a non-broken version (which for Celery, is 5.3.1+).
3. Or error, but do so with a helpful error message (explaining broken packages are installed and to either manually remove those packages or downgrade pip), rather than show an unhandled internal error/traceback.

pip version

24.2

Python version

3.12.5 (homebrew)

OS

macOS

How to Reproduce

1. mkdir testcase && cd $_
2. python -m venv .venv && source .venv/bin/activate
3. pip install pip==24.0
4. pip install celery==4.4.7
5. pip install pip==24.2
6. pip install celery

Output

(.venv) testcase $ pip install celery
Requirement already satisfied: celery in ./.venv/lib/python3.12/site-packages (4.4.7)
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_vendor/packaging/requirements.py", line 36, in __init__
    parsed = _parse_requirement(requirement_string)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_vendor/packaging/_parser.py", line 62, in parse_requirement
    return _parse_requirement(Tokenizer(source, rules=DEFAULT_RULES))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_vendor/packaging/_parser.py", line 80, in _parse_requirement
    url, specifier, marker = _parse_requirement_details(tokenizer)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_vendor/packaging/_parser.py", line 118, in _parse_requirement_details
    specifier = _parse_specifier(tokenizer)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_vendor/packaging/_parser.py", line 208, in _parse_specifier
    with tokenizer.enclosing_tokens(
  File "/opt/homebrew/Cellar/python@3.12/3.12.5/Frameworks/Python.framework/Versions/3.12/lib/python3.12/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_vendor/packaging/_tokenizer.py", line 189, in enclosing_tokens
    self.raise_syntax_error(
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_vendor/packaging/_tokenizer.py", line 167, in raise_syntax_error
    raise ParserSyntaxError(
pip._vendor.packaging._tokenizer.ParserSyntaxError: Expected matching RIGHT_PARENTHESIS for LEFT_PARENTHESIS, after version specifier
    pytz (>dev)
         ~^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 105, in _run_wrapper
    status = _inner_run()
             ^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 96, in _inner_run
    return self.run(options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 67, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_internal/commands/install.py", line 379, in run
    requirement_set = resolver.resolve(
                      ^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
    result = self._result = resolver.resolve(
                            ^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 427, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 239, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 229, in _get_updated_criteria
    for requirement in self._p.get_dependencies(candidate=candidate):
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 247, in get_dependencies
    return [r for r in candidate.iter_dependencies(with_requires) if r is not None]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 401, in iter_dependencies
    for r in self.dist.iter_dependencies():
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_dists.py", line 215, in iter_dependencies
    req = get_requirement(req_string.strip())
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_internal/utils/packaging.py", line 45, in get_requirement
    return Requirement(req_string)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/emorley/src/testcase/.venv/lib/python3.12/site-packages/pip/_vendor/packaging/requirements.py", line 38, in __init__
    raise InvalidRequirement(str(e)) from e
pip._vendor.packaging.requirements.InvalidRequirement: Expected matching RIGHT_PARENTHESIS for LEFT_PARENTHESIS, after version specifier
    pytz (>dev)
         ~^

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[edmorley]

This blocks Heroku upgrading from pip 24.0 to something newer (eg heroku/heroku-buildpack-python#1619), since we will no doubt have a long tail of users with existing cached site-packages with packages using non-standard dependency specifiers installed. (Whilst users should be upgrading to newer versions of packages with those specifiers fixed, if there isn't a clear error message, they won't know what the issue is or how to resolve it.)

Note: This issue seems similar to #12945, though that issue seems more about initial installation, rather than "existing installation", plus doesn't have explicit steps to reproduce - so it seemed best to file a separate issue.

[notatallshaw]

I've created #12953 that gives a much clearer error message:

$ python -m pip install --dry-run celery
Requirement already satisfied: celery in ./.venv/lib/python3.12/site-packages (4.4.7)
error: invalid-installed-package

× Cannot process installed package celery 4.4.7 in '/home/damian/opensource/contributions/pip/.venv/lib/python3.12/site-packages' because it has an invalid requirement:
│ Expected matching RIGHT_PARENTHESIS for LEFT_PARENTHESIS, after version specifier
│     pytz (>dev)
│          ~^
╰─> Starting with pip 24.1, packages with invalid requirements can not be processed.

hint: To proceed this package must be uninstalled using 'pip<24.1', some other Python package tool, or manually deleted.

Edit: Updated message to reflect latest version of PR