Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FAQ: Clarify poetry2nix does not use nixpkgs for Python packages #1193

Merged
merged 2 commits into from
Jul 1, 2023

Conversation

asymmetric
Copy link
Contributor

I'm sure this could be expanded upon, and maybe this PR can provide the impetus for that.

@takeda
Copy link
Contributor

takeda commented Jun 28, 2023

This is not strictly true. Many packages that aren't explicitly defined in the lock file are actually obtained from nixpkgs.

This mostly applies to overrides, things like when package needs setuptools, flit, poetry, hatch and of course non python dependencies like openssl etc.

@asymmetric
Copy link
Contributor Author

asymmetric commented Jun 29, 2023

Interesting, I wrote this up after a comment by @adisbladis, but I might've interpreted it too broadly.

But in the end, this was my goal with this PR, to help spell out the architecture so we can document it somewhere visible.

Many packages that aren't explicitly defined in the lock file are actually obtained from nixpkgs.

@takeda can you point to locations where this is handled?

@takeda
Copy link
Contributor

takeda commented Jun 29, 2023

I see. The context matters. The question was about package (pillow) listed as a dependency to author's application. In that context pillow defined in poetry is used after applying overrides. The nixpkgs version is totally tossed away.

But then to install pillow poetry2nix adds setuptools dependency:

"pillow": [
"setuptools"
],

That dependency is not defined in poetry.lock file and is pulled from nixpkgs.

You might think "oh this is a special case and it only happens at build time", but then there are broken packages like ansible, where all its dependencies are pulled from nixpkgs because it doesn't define any:

ansible = super.ansible.overridePythonAttrs (
old: {
# Inputs copied from nixpkgs as ansible doesn't specify it's dependencies
# in a correct manner.
propagatedBuildInputs = (old.propagatedBuildInputs or [ ]) ++ [
self.pycrypto
self.paramiko
self.jinja2
self.pyyaml
self.httplib2
self.six
self.netaddr
self.dnspython
self.jmespath
self.dopy
self.ncclient
];
}
);

So the answer to this question is much more complex.

@asymmetric
Copy link
Contributor Author

asymmetric commented Jun 30, 2023

So a better answer is that everything declared in a poetry.lock is built without relying on nixpkgs definitions, with the exception of under-specified packages, like e.g. ansible, or build backends, as specified in build-systems.json.

Is this accurate?

@takeda
Copy link
Contributor

takeda commented Jul 1, 2023

The thing is that what is clear to me often is not clear to others, but this is what I would write:

poetry2nix overlays packages taken from poetry.lock on top of nixpkgs (in such way that overwritten packages in nixpkgs are completely ignored). Any package that is used, that isn't in the poetry.lock file (most commonly build dependencies) is taken from nixpkgs.

@adisbladis adisbladis merged commit 215afa1 into nix-community:master Jul 1, 2023
108 checks passed
@asymmetric asymmetric deleted the patch-1 branch July 1, 2023 17:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants