-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[red-knot] Add basic workspace support #12318
Conversation
ec29832
to
d0b9e16
Compare
d0b9e16
to
a6bdb89
Compare
a6bdb89
to
49cbacc
Compare
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The core abstractions here make a lot of sense to me! This is a great step forward in clarifying how everything will fit together.
49cbacc
to
7bb31d1
Compare
7562aaf
to
146a054
Compare
146a054
to
f1696e5
Compare
Summary
This PR adds support for checking an entire workspace rather than just individual files. But it does more than that. It lays the foundation for how Red Knot integrates settings long-term.
The CLI changes from
red_knot <files>
to something more cargo like where runningred_knot
defaults to checking the entire workspace in the current directory. Right now, any directory is a valid workspace and each workspace always has exactly one package. The long-term goal is to find the enclosing workspace by reading Ruff's configuration (one with a[workspace]
section?) and then finding all packages specified in[workspace.members]
.New
Workspace
andPackage
ingredientsThese ingredients represent the project's structure. They aren't used in any queries today, but I foresee the following usages:
package.check
: Checks all files in the package. We could make this a query to short-circuit the analysis. This should be especially useful in projects with many packages.package.files
andpackage.settings
: See below for the details on how we can compute thefiles
andsettings
lazily.This PR also introduces
WorkspaceMetadata
andPackageMetdata
struct. These are the equivalent ofWorkspace
andPackage
but as non-Salsa ingredient. The key motivation for these two is that we need to resolve thetarget-version
and the module-resolution settings to decide on the persistent cache to load. This requires access to the workspace's and package's settings, and that's what these two structs will expose in the future.Should
Workspace
be a singleton?Possibly? I'm not sure. The idea is that a
Database
has exactly oneWorkspace
. To me it is unclear if it may be necessary to create a secondWorkspace
in response to file changes. E.g. what if theworkspace
'sruff.toml
gets deleted? Maybe we should just create a newDatabase
in that case but that complicates things in the CLI's main loop. My current thinking is to create a newWorkspace
ingredient in that case (which would fail if it is a singleton) and swap theProgram
'sWorkspace
. This should also allow us to keep the existingWorkspace
if loading the new one, for whatever reason, fails.New
Program
ingredientThe new
Program
ingredient (singleton) mainly generalizes the existing module resolution settings and moves them intoruff_db
. The motivation is that these settings aren't just important for module resolution. They're fundamental to the entire analysis. Two analyses using different program settings aren't compatible and should be cached (persistent) separately.@AlexWaygood we have to figure out how to solve our merge conflicts.
I decided to rename the root
Database
fromProgram
toRootDatabase
, mainly because I found it counterintuitive that a program contains aWorkspace
and to avoid a naming conflict with the newProgram
ingredient.Should the
Database
support multiple openWorkspace
s?My initial thinking was that it probably should, but I've concluded that it would complicate persistent caching by a lot.
Ideally, Ruff uses a different persistent cache for different workspaces. Storing multiple
Workspace
s in a singleDatabase
creates the challenge that it becomes necessary to decide for everyIngredient
into which persistent cache it should be stored. Loading the persistent cache would require merging theIngredient
s from different caches, which seems a headache.Having a single
Database
perProgramSettings
also simplifies caching runs using different target-versions individually:ruff check --target-version 3.8
andruff check --target-version 3.10
should each use their own persistent cache.For now, I think it's best to have a 1:1 relationship between
Database
andWorkspace
. This doesn't mean Red Knot can't support multiple openWorkspace
s. It just means we must build this functionality around theDatabase
instead of into theDatabase
.Checing individual files or folders
It remains possible to check a different folder using
red_knot—-current-directory path
. However, this PR removes the functionality for testing specific files or sub-folders. The challenge with explicit paths is that it isn't sufficient to detect the files in these paths once; instead, the paths must be monitored for newly added or removed files (and follow Ruff's exclude semantics). I'm not sure how to implement this best, but I think this is a CLI-specific problem and should be built on the addedopen_files
feature.How do settings work?
This is about the formatter or linter settings, not the workspace or package settings. Ruff supports that each directory can have its own linter and formatter settings. That means it is necessary to resolve the settings on a directory level.
The easiest solution is to define a
linter_settings(File)
query that returns the liner settings for a given file by searching the closest package and asking that package what its settings are. The main downside is that the query is more granular than what Ruff allows (it's a file-level query rather than a directory-level query). A possible solution is to create a newDirectory
ingredient (interned) and change the query tolinter_settings(Directory)
. While this query now has the correct granularity, it still means we're creating too many ingredients if a project has only a custom setting in exactly two directories.I think the best solution is to instead create a
Settings
input ingredient that has trackedlinter_settings
,formatter_settings
, etc. methods. A package would create all theSettings
the first time it is asked for a setting file (we can find all the settings while discovering all the package files). Resolving the linter settings for a file then simply becomes:Settings
(can use a router similar to Ruff today)settings.linter_settings
(salsa query)Dependent queries would only re-run when:
That's pretty good.
Can
packages.files
andpackages.settings
be lazy?In this PR, discovering the files of a package is done eagerly. I think it should be possible to make this lazily but i would prefer to do this in a separate PR. The overall idea is:
SourceRoot
input ingredient. A source root is a directory, but it's not a 1:1 mapping betweenSourceRoot
and a directory on the disk. A source root has two fields: Its a path (id) and arevision.
SourceRoot
for theWorkspace
and everyPackage
revision
depends on the specificSourceRoot
max
:ph
file ([red-knot] Add support for editable installs to the module resolver #12307)Edit: I think we can simplify this by simple storing an incrementing counter on
Package
andWorkspace
rather then introducing a new ingredient.Follow up work
Test Plan
I ran
red_knot --current-directory ../test
and verified that the program is re-checked when adding, removing, or deleting files.