[red-knot] Add support for `--system-site-packages` virtual environments #12759

AlexWaygood · 2024-08-08T16:32:35Z

Summary

Many things about Python virtual environments are not specified, but the following things are:

All Python virtual environments have a pyvenv.cfg file in the venv root.
The pyvenv.cfg file is guaranteed to have a home = <path> key in it, which will point to the parent directory of the system Python installation that was used to create this venv
The venv will always have its own, isolated site-packages directory which will be appended to sys.path at runtime by the stdlib site module
If the pyvenv.cfg file contains include-system-site-packages = true, (or include-system-site-packages = TrUe, or any other case variations of the word "true"), Python's stdlib module will also append the system Python's site-packages directory as well as the venv's isolated site-packages directory.

This PR emulates that runtime behaviour in red-knot.

Test Plan

Several tests have been added to this PR.

The pyvenv.cfg files provided by uv, virtualenv and the stdlib venv module are all consistent on the above bullets, but differ in many other subtle aspects. As such, I've added two more venvs to the resources/test directory, in addition to the uv-created one: one created using virtualenv and one created using the stdlib venv module.

I haven't added any tests with venvs that actually have include-system-site-packages = true in their pyvenv.cfg files. I'm not sure how I'd add tests for that without actually committing a complete system Python installation into the resources/test directory. I did manual testing with --system-site-packages venvs created using all three tools, however, and everything seemed to be working as expected. (I tried using both a pyenv-installed Python and a homebrew-installed Python.)

MichaReiser · 2024-08-08T16:44:28Z

crates/red_knot_workspace/src/site_packages.rs

Should this now be moved into the semantic index crate?

at some point, yes

crates/red_knot_workspace/src/site_packages.rs

MichaReiser

Looks exciting. I'll take a closer look tomorrow

crates/red_knot_workspace/src/site_packages.rs

AlexWaygood · 2024-08-08T18:14:16Z

Ah dang. The existing tests that I previously added in 7fa76a2 now fail, because we now reject a venv as an invalid venv if the venv's pyvenv.cfg file contains a home key that doesn't point to an actually existing directory. And the virtual environments that I committed to resources/test all have home keys in their pyvenv.cfg files that point to directories that exist for me locally, but that don't exist in CI (because I have not committed my system Python installation to the Ruff repo).

Not sure what to do here... it would be really nice if we could have some way of creating a venv on the fly in the context of our test suite... but the venv would need to depend on some system Python installation, which would mean that our test suite would fail unless you had Python installed...

I guess we could just give up on the idea of testing with "real" venvs and create mock venv-like directory structures in the test? But it still feels unclear to me how valuable a test like that would actually be, since we'd really just be repeating the logic from the implementation in the test

crates/red_knot_workspace/src/site_packages.rs

carljm

Looks good!

I guess we could just give up on the idea of testing with "real" venvs and create mock venv-like directory structures in the test? But it still feels unclear to me how valuable a test like that would actually be, since we'd really just be repeating the logic from the implementation in the test

I know we touched on this briefly in the previous PR, but I really think this is a non-issue, and having tests that use (accurate) "mock" venvs is very useful, and in fact is precisely equally useful to using "real" venvs that we've committed directly.

Once we commit the files into our repo, it is for all intents and purposes a mock either way, because we don't have any ongoing validation that it continues to match what the venv module (or whatever other venv creator) does, and because the tests aren't running on the real system where the venv was created and can actually function. It's good to base our mocks carefully on real observed venvs, but it's also totally fine to then strip out known-to-be irrelevant parts, change the home path to point to an actual directory, etc. None of that makes the tests any less useful.

If it were me, I'd remove a bunch more unnecessary cruft from the committed mock venvs (like pip scripts and whatnot that are also non-functional because they refer to non-existent directories.)

I think the home thing will require (at least partly) switching from pre-committed mocks to written-out-by-the-test mocks, because it should be an absolute path (changing it to a relative path would IMO be too much of a departure from a real venv), and you won't know in advance what absolute path you can write to on some arbitrary system the tests run on.

carljm · 2024-08-08T19:56:18Z

crates/red_knot_workspace/src/site_packages.rs

+    }
+
+    fn from_executable_home_path(path: &PythonHomePath, system: &dyn System) -> Option<Self> {
+        let candidate = path.parent()?;


FWIW I don't think this is true for Windows. I don't have a Windows machine handy to check, but I have a vague memory (from many years ago) that on Windows the Python binary goes directly in the prefix dir of the installation, not inside \Scripts, which is the Windows equivalent to the bin/ dir.

(This may also explain the lack of clarity in the docs around bin/ dir vs prefix dir; IIRC Vinay, who wrote the docs, uses Windows, where I think there is no distinction to be made.)

crates/red_knot_workspace/src/site_packages.rs

MichaReiser

This is great.

I mainly left questions around how we handle errors and how to handle relative paths.

I'm not too fond of PythonMinorVersion. I still think we should just use TargetVersion

MichaReiser · 2024-08-09T07:05:49Z

crates/red_knot/src/main.rs

@@ -9,7 +9,7 @@ use salsa::plumbing::ZalsaDatabase;

 use red_knot_server::run_server;
 use red_knot_workspace::db::RootDatabase;
-use red_knot_workspace::site_packages::site_packages_dirs_of_venv;
+use red_knot_workspace::site_packages::VirtualEnvironment;


It feels a bit surprising that VirtualEnvironment is part of the site_packages module. Shouldn' we rename the site_packages module to venv?

That might make @carljm upset, since he keeps pointing out that several of the routines in this module really work with any Python installation, not just virtual environments 😆

Hmm yeah. Not sure. The hierarchy just feels slightly off to me. But we can also revisit this later.

Well, it would depend how you do it :) I was actually going to make a similar comment and decided not to. But I would not want to just rename this module, I would want two modules, one which is specific to venvs and one which contains stuff that works on Python installations in general. But I'm also fine with just keeping it all in one module for now, which is why I didn't make that comment :)

Okay, I'll leave this for now; we can come back to it later

crates/red_knot/src/main.rs

crates/red_knot_workspace/src/site_packages.rs

… search path if `include-system-site-packages` is specified

github-actions · 2024-08-09T17:09:12Z

`ruff-ecosystem` results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

AlexWaygood · 2024-08-09T18:30:40Z

I think I've now addressed everything except for https://github.com/astral-sh/ruff/pull/12759/files#r1711881269. Would appreciate it if somebody could take another quick look, though, as it's changed quite a lot today! In particular, the tests have been completely reworked (which was necessary to make them parse): there's no longer any committed virtual environments; we just create small mocks in tests.

carljm

Looks good to me!

carljm · 2024-08-09T19:26:02Z

crates/red_knot_workspace/src/site_packages.rs

+        if cfg!(target_os = "windows") {
+            Some(Self(path.to_path_buf()))


Were you able to verify this, or just trusted my vague years-old memory here? 😆

I verified it on an extremely slow, very old Windows machine I had lying around. I had to install Python on the machine. It took forever -_-

carljm · 2024-08-09T19:30:14Z

crates/red_knot_workspace/src/site_packages.rs

+                if cfg!(target_os = "windows") {
+                    let system_home_path = system_install_sys_prefix.clone();
+                    let system_exe_path = system_home_path.join("python.exe");
+                    let system_site_packages_path =
+                        system_install_sys_prefix.join(r"Lib\site-packages");
+                    (system_home_path, system_exe_path, system_site_packages_path)


Related to my question above: I think if it hasn't happened already and doesn't happen before merge, we should track a follow-up task for someone with access to a Windows machine to actually audit this and make sure we are testing for behaviors that actually match what a Windows system / venv looks like.

agreed, these tests really need to be supplemented with manual testing

Manual testing would be great, but if you at least validated that this mock setup matches what you see on a real Windows machine with Python installed on it, that's a great start for now.

Yeah, I did that much. But considering how long installing Python took, there's no way I'm installing the Rust toolchain and doing manual testing of this PR on that machine 😆

carljm · 2024-08-09T19:31:14Z

crates/red_knot_workspace/src/site_packages.rs

+            let unix_site_packages = if *free_threaded {
+                format!("lib/python3.{minor_version}t/site-packages")
+            } else {
+                format!("lib/python3.{minor_version}/site-packages")
+            };


Any particular reason we calculate this unconditionally out here, rather than in the non-Windows block below?

It's just annoying to have to define such a long variable so many times over and over... and this is a test, so I don't think binary size really matters quite so much here :P

AlexWaygood requested review from carljm and MichaReiser as code owners August 8, 2024 16:32

AlexWaygood added the red-knot Multi-file analysis & type inference label Aug 8, 2024

MichaReiser reviewed Aug 8, 2024

View reviewed changes

crates/red_knot_workspace/src/site_packages.rs Outdated Show resolved Hide resolved

MichaReiser reviewed Aug 8, 2024

View reviewed changes

crates/red_knot_workspace/src/site_packages.rs Outdated Show resolved Hide resolved

MichaReiser reviewed Aug 8, 2024

View reviewed changes

crates/red_knot_workspace/src/site_packages.rs Outdated Show resolved Hide resolved

crates/red_knot_workspace/src/site_packages.rs Outdated Show resolved Hide resolved

crates/red_knot_workspace/src/site_packages.rs Outdated Show resolved Hide resolved

AlexWaygood force-pushed the alex/system-site-packages branch from fd049a8 to 18a7eaf Compare August 8, 2024 18:23

AlexWaygood commented Aug 8, 2024

View reviewed changes

crates/red_knot_workspace/src/site_packages.rs Show resolved Hide resolved

carljm approved these changes Aug 8, 2024

View reviewed changes

MichaReiser approved these changes Aug 9, 2024

View reviewed changes

AlexWaygood mentioned this pull request Aug 9, 2024

[red-knot] Move, rename and make public the PyVersion type #12782

Merged

AlexWaygood force-pushed the alex/system-site-packages branch 2 times, most recently from c414304 to a262da6 Compare August 9, 2024 13:40

AlexWaygood added 6 commits August 9, 2024 16:51

Implement parsing for pyvenv.cfg files

1b47232

Use pyvenv.cfg files to add system site-packages as an additional…

87c0fd6

… search path if `include-system-site-packages` is specified

Address review

61c9036

Address many review comments

40b9ea8

Improve pyvenv.cfg parsing

e213bb1

Add a comment

08eac4f

AlexWaygood force-pushed the alex/system-site-packages branch from bf6ebf5 to 1e0c08b Compare August 9, 2024 16:42

rework the testing strategy

d4c933a

AlexWaygood force-pushed the alex/system-site-packages branch from 1e0c08b to d4c933a Compare August 9, 2024 16:44

AlexWaygood added 2 commits August 9, 2024 17:55

Use PythonVersion from red_knot_python_semantic

fe1ceb1

Maybe fix Windows?

535a68d

AlexWaygood added 3 commits August 9, 2024 18:21

error handling improvements

bc86a4e

More docs for PythonHomePath

27099e8

fix Windows?

6ffb8c7

Resolve --venv-path relative to cli_base_path in the CLI

b8da0cf

AlexWaygood requested review from carljm and MichaReiser August 9, 2024 18:32

carljm approved these changes Aug 9, 2024

View reviewed changes

AlexWaygood changed the title ~~[red-knot] Use the system Python's site-packages as an additional search path if include-system-site-packages = true is present in a virtual environment's pyvenv.cfg file~~ [red-knot] Add support for --system-site-packages virtual environments Aug 9, 2024

AlexWaygood merged commit 37b9bac into main Aug 9, 2024
20 checks passed

AlexWaygood deleted the alex/system-site-packages branch August 9, 2024 20:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[red-knot] Add support for `--system-site-packages` virtual environments #12759

[red-knot] Add support for `--system-site-packages` virtual environments #12759

AlexWaygood commented Aug 8, 2024

MichaReiser Aug 8, 2024 •

edited

Loading

AlexWaygood Aug 8, 2024

MichaReiser left a comment

AlexWaygood commented Aug 8, 2024

carljm left a comment •

edited

Loading

carljm Aug 8, 2024 •

edited

Loading

MichaReiser left a comment

MichaReiser Aug 9, 2024

AlexWaygood Aug 9, 2024

MichaReiser Aug 9, 2024

carljm Aug 9, 2024

AlexWaygood Aug 9, 2024

github-actions bot commented Aug 9, 2024 •

edited

Loading

AlexWaygood commented Aug 9, 2024 •

edited

Loading

carljm left a comment

carljm Aug 9, 2024

AlexWaygood Aug 9, 2024 •

edited

Loading

carljm Aug 9, 2024

AlexWaygood Aug 9, 2024

carljm Aug 9, 2024

AlexWaygood Aug 9, 2024

carljm Aug 9, 2024

AlexWaygood Aug 9, 2024

		if cfg!(target_os = "windows") {
		Some(Self(path.to_path_buf()))

[red-knot] Add support for --system-site-packages virtual environments #12759

[red-knot] Add support for --system-site-packages virtual environments #12759

Conversation

AlexWaygood commented Aug 8, 2024

Summary

Test Plan

MichaReiser Aug 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MichaReiser left a comment

Choose a reason for hiding this comment

AlexWaygood commented Aug 8, 2024

carljm left a comment • edited Loading

Choose a reason for hiding this comment

carljm Aug 8, 2024 • edited Loading

Choose a reason for hiding this comment

MichaReiser left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Aug 9, 2024 • edited Loading

ruff-ecosystem results

Linter (stable)

Linter (preview)

AlexWaygood commented Aug 9, 2024 • edited Loading

carljm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlexWaygood Aug 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

[red-knot] Add support for `--system-site-packages` virtual environments #12759

[red-knot] Add support for `--system-site-packages` virtual environments #12759

MichaReiser Aug 8, 2024 •

edited

Loading

carljm left a comment •

edited

Loading

carljm Aug 8, 2024 •

edited

Loading

github-actions bot commented Aug 9, 2024 •

edited

Loading

`ruff-ecosystem` results

AlexWaygood commented Aug 9, 2024 •

edited

Loading

AlexWaygood Aug 9, 2024 •

edited

Loading