Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow pipenv option to consider all listed package sources when syncing the Pipfile.lock #5039

Merged
merged 7 commits into from
Apr 21, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion docs/advanced.rst
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,19 @@ as sub-dependencies must be mirrored onto your private index or they will not re
standard recommendation of ``pip`` maintainers: "To correctly make a private project installable is to point
--index-url to an index that contains both PyPI and their private projects—which is our recommended best practice."

The above documentation holds true for both ``lock`` resolution and ``sync`` of packages. It was suggested that
once the resolution and the lock file are updated, it is theoretically possible to safely scan multiple indexes
for these packages when running ``pipenv sync`` or ``pipenv install --deploy`` since it will verify the package
hashes match the allowed hashes that were already captured from a safe locking cycle.
To enable this non-default behavior, add ``install_search_all_sources = true`` option
to your ``Pipfile`` in the ``pipenv`` section::

[pipenv]
install_search_all_sources = true

**Note:** The locking cycle will still requires that each package be resolved from a single index. This feature was
requested as a workaround in order to support organizations where not everyone has access to the package sources.

☤ Using a PyPI Mirror
----------------------------

Expand Down Expand Up @@ -119,7 +132,7 @@ If you want to work with private registries that use the keychain for authentica
can disable the "enforcement of no input".

**Note:** Please be sure that the keychain will really not ask for
input. Otherwise the process will hang forever!
input. Otherwise the process will hang forever!::

[[source]]
url = "https://pypi.org/simple"
Expand Down
2 changes: 2 additions & 0 deletions news/5041.feature.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Added new Pipenv option ``install_search_all_sources`` that allows installation of packages from an
existing ``Pipfile.lock`` to search all defined indexes for the constrained package version and hash signatures.
28 changes: 15 additions & 13 deletions pipenv/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -1466,10 +1466,8 @@ def pip_install(
use_pep517=True,
):
piplogger = logging.getLogger("pipenv.patched.notpip._internal.commands.install")
src_dir = None
if not trusted_hosts:
trusted_hosts = []

trusted_hosts.extend(os.environ.get("PIP_TRUSTED_HOSTS", []))
if not allow_global:
src_dir = os.getenv(
Expand All @@ -1484,22 +1482,26 @@ def pip_install(
ignore_hashes = False
line = None
# Try installing for each source in project.sources.
search_all_sources = project.settings.get("install_search_all_sources", False)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add public facing documentation for this option?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes -- I am going to add that once we decide if this should be what its name is.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the name install_search_all_sources is OK. Once you write the docs for this, you can figure if this works or not.

Copy link
Member Author

@matteius matteius Apr 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Squeezing this into tonight's release -- let me know if there are any follow up tweaks you want to see to the docs or this PR and I can get them in a subsequent PR.

if not index and requirement.index:
index = requirement.index
if index and not extra_indexes:
extra_indexes = []
if requirement.index:
extra_indexes = list(
filter(lambda d: d.get("name") == requirement.index, project.sources)
)
if not extra_indexes:
if search_all_sources:
extra_indexes = list(project.sources)
else: # Default: index restrictions apply during installation
extra_indexes = []
if requirement.index:
extra_indexes = list(
filter(lambda d: d.get("name") == requirement.index, project.sources)
)
if not extra_indexes:
extra_indexes = list(project.sources)
if requirement and requirement.vcs or requirement.editable:
requirement.index = None
# Install dependencies when a package is a non-editable VCS dependency.
# Don't specify a source directory when using --system.
if not requirement.editable and no_deps is not True:
# Leave this off becauase old lockfiles don't have all deps included
# Leave this off because old Pipfile.lock don't have all deps included
# TODO: When can it be turned back on?
no_deps = False
elif requirement.editable and no_deps is None:
Expand All @@ -1519,7 +1521,7 @@ def pip_install(
trusted_hosts=trusted_hosts,
pypi_mirror=pypi_mirror,
)
if requirement.index in sources:
if not search_all_sources and requirement.index in sources:
sources = list(filter(lambda d: d.get("name") == requirement.index, sources))
if r:
with open(r, "r") as fh:
Expand Down Expand Up @@ -1558,10 +1560,10 @@ def pip_install(
if project.s.is_verbose():
click.echo(f"$ {cmd_list_to_shell(pip_command)}", err=True)
cache_dir = Path(project.s.PIPENV_CACHE_DIR)
DEFAULT_EXISTS_ACTION = "w"
default_exists_action = "w"
if selective_upgrade:
DEFAULT_EXISTS_ACTION = "i"
exists_action = project.s.PIP_EXISTS_ACTION or DEFAULT_EXISTS_ACTION
default_exists_action = "i"
exists_action = project.s.PIP_EXISTS_ACTION or default_exists_action
pip_config = {
"PIP_CACHE_DIR": cache_dir.as_posix(),
"PIP_WHEEL_DIR": cache_dir.joinpath("wheels").as_posix(),
Expand Down