Sub dependencies not being installed when explicitly providing a repository via source

[svbecker19]

Description

When specifying a source in the pyproject.toml and set the priority to default or primary on package installation certain sub dependencies will not be installed for some reason. By not installing those dependencies I get a broken environment with ModuleNotFound errors all over the place.

For example when trying to install the package 'opentelemetry-api' only the package will get installed but not its dependencies like 'importlib-metadata', 'deprecated' etc..

However testing the same without a source block (or with some other priority) the mentioned dependencies will get installed.

Workarounds

There is no workaround as far as I know when you need a private repository.

Poetry Installation Method

install.python-poetry.org

Operating System

MacOS

Poetry Version

Poetry (version 1.8.2)

Poetry Configuration

cache-dir = "/Users/svbecker/Library/Caches/pypoetry"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
keyring.enabled = true
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"
virtualenvs.prefer-active-python = true
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true

Python Sysconfig

Platform: "macosx-13.2-arm64"
Python version: "3.11"
Current installation scheme: "venv"

Example pyproject.toml

[tool.poetry]
name = "dummy"
version = "0.1.0"
description = ""
authors = []

[tool.poetry.dependencies]
python = "^3.11"
opentelemetry-api = "^1.24.0"

[[tool.poetry.source]]
name = "pypi-primary"
url = "https://pypi.org/simple"
priority = "primary"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Poetry Runtime Logs

❯ poetry add -vvv opentelemetry-api
Loading configuration file /Users/svbecker/Library/Application Support/pypoetry/config.toml
Adding repository pypi-primary (https://pypi.org/simple) and setting it as primary
Deactivating the PyPI repository
Trying to detect current active python executable as specified in the config.
Found: /Users/svbecker/.local/share/pyenv/versions/3.11.1/bin/python
Using virtualenv: /Users/svbecker/Workspace/projects/dummy/.venv
Checking if keyring is available
[keyring:keyring.backend] Loading KWallet
[keyring:keyring.backend] Loading SecretService
[keyring:keyring.backend] Loading Windows
[keyring:keyring.backend] Loading chainer
[keyring:keyring.backend] Loading libsecret
[keyring:keyring.backend] Loading macOS
Using keyring backend 'macOS Keyring'
Creating new session for pypi.org
Source (pypi-primary): 31 packages found for opentelemetry-api *
Using version ^1.24.0 for opentelemetry-api

Updating dependencies
Resolving dependencies...
   1: fact: dummy is 0.1.0
   1: derived: dummy
   1: fact: dummy depends on opentelemetry-api (^1.24.0)
   1: selecting dummy (0.1.0)
   1: derived: opentelemetry-api (>=1.24.0,<2.0.0)
Source (pypi-primary): 1 packages found for opentelemetry-api >=1.24.0,<2.0.0
   1: selecting opentelemetry-api (1.24.0)
   1: Version solving took 0.002 seconds.
   1: Tried 1 solutions.

Finding the necessary packages for the current system
Source (pypi-primary): 1 packages found for opentelemetry-api >=1.24.0,<2.0.0

Package operations: 1 install, 0 updates, 0 removals

  - Installing opentelemetry-api (1.24.0): Pending...
  - Installing opentelemetry-api (1.24.0): Installing...
  - Installing opentelemetry-api (1.24.0)

Writing lock file

[alban-bnch]

We are seeing a similar issue. In our case, it's httpcore not installing it's sub-dependencies. We have seen this happen with both poetry 1.7.1 and poetry 1.8.2.

A few notes if it helps:

1. clearing the poetry cache entirely did not help.
2. the issue happens only when using a custom source.
3. after clearing the poetry cache and installing httpcore with a custom source, only the httpcore package is added to the project and in the poetry.lock. When clearing the cache again, it looks like we have 3 packages there (which matches httpcore and its 2 sub-dependencies).
4. the issue happened both locally (macos) and in CI.
5. the issue happened on a template repo which does not have poetry.lock committed.
6. on an existing repo, running poetry update does remove the sub-dependencies of httpcore.

Hope this helps.

[alban-bnch]

Also, this is most likely related to #9266 (comment)

[dimbleby]

yes this will be duplicate #9130, #9191, #9195, #9244, #9266, #9272, #9288, and possibly more that I have missed

please close

[svbecker19]

Thanks for pointing out the related issues.

I did the following steps in some order I don't remember but it is working now:

• Clear poetry cache (not sure which one, thus I cleared them all)
• poetry self add pkginfo==1.10.0
• poetry self add keyrings.cryptfile@latest (not sure why but otherwise the poetry self install command failed)
• poetry self lock && poetry self install (not sure if needed)

Are those the recommended steps to solve this issue? If so I assume this is a workaround and will be fixed in the next bugfix/minor version?! Or does every developer or CI server using poetry needs to do the steps described above (or use pipx instead of the installer script?)?

[dimbleby]

this all is covered in the existing issues

[david-waterworth]

poetry self add keyrings.cryptfile@latest downgraded pkginfo

poetry self add keyrings.cryptfile@latest
Using version ^1.3.9 for keyrings-cryptfile

Updating dependencies
Resolving dependencies... (3.5s)

Package operations: 4 installs, 5 updates, 9 removals

  • Removing attrs (23.1.0)
  • Removing html5lib (1.1)
  • Removing jsonschema (4.19.0)
  • Removing jsonschema-specifications (2023.7.1)
  • Removing lockfile (0.12.2)
  • Removing referencing (0.30.2)
  • Removing rpds-py (0.9.2)
  • Removing six (1.16.0)
  • Removing webencodings (0.5.1)
  • Updating rapidfuzz (3.5.2 -> 3.9.0)
  • Installing argon2-cffi-bindings (21.2.0)
  • Updating build (1.0.3 -> 1.2.1)
  • Updating fastjsonschema (2.18.1 -> 2.19.1)
  • Updating keyring (24.2.0 -> 24.3.1)
  • Downgrading pkginfo (1.10.0 -> 1.9.6)
  • Installing argon2-cffi (23.1.0)
  • Installing pycryptodome (3.20.0)
  • Installing keyrings-cryptfile (1.3.9)

I'm not sure that step is necessary, clearing the cache and upgrading pkginfo seems sufficient

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.