[red-knot] Add walk_directories to System

[MichaReiser]

Summary

This PR adds a ignore::WalkDir like API to System that allows walking a directory tree.

We'll need this to implement the analysis of an entire directory.

Test Plan

Added unit tests.

[codspeed-hq]

CodSpeed Performance Report

Merging #12297 will not alter performance

_{Comparing system-walk-directories (7cfe17b) with system-walk-directories (b22ac5e)}

Summary

✅ 33 untouched benchmarks

[MichaReiser]

I'm happy to revert this pack to returning an impl. I had to change it, because a first version of the walker had to store the current iteration state in a struct which doesn't work with impl (Unless you use dyn). I introduced the named type and thought I'll leave it, because it can be useful.

It's also nice that the type now implements Debug

[github-actions]

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

ℹ️ ecosystem check encountered format errors. (no format changes; 1 project error)

openai/openai-cookbook (error)

warning: Detected debug build without --no-cache.
error: Failed to parse examples/chatgpt/gpt_actions_library/.gpt_action_getting_started.ipynb:11:1:1: Expected an expression
error: Failed to parse examples/chatgpt/gpt_actions_library/gpt_action_bigquery.ipynb:13:1:1: Expected an expression

Formatter (preview)

ℹ️ ecosystem check encountered format errors. (no format changes; 1 project error)

openai/openai-cookbook (error)

ruff format --preview

warning: Detected debug build without --no-cache.
error: Failed to parse examples/chatgpt/gpt_actions_library/.gpt_action_getting_started.ipynb:11:1:1: Expected an expression
error: Failed to parse examples/chatgpt/gpt_actions_library/gpt_action_bigquery.ipynb:13:1:1: Expected an expression

[carljm]

It seems like most places we prefer having enum variants wrap independent named structs over struct enum variants. I'm curious if you have thoughts on when to choose one vs the other.

[MichaReiser]

To me it depends on how the enum variants are used. Our AST used to use struct enum variants over named structs. This was very cumbersome in downstream tools because it was impossible to pass an entire StmtFunctionDef, because it lacked a name. That's why you still see many lint rules accepting all individual fields of their node instead of the entire node.

Another motivation for named structs is if you want to lock-down access to individual fields because the enum fields always have the same visibility as the enum itself.

On the other hand, using variant structs allows Rust to be more clever when it comes to layout optimizations and it's a lot less boilerplate to define the enum and to match on it (no nesting).

I'm using struct variants here because

a) I think it's unlikely that some code would want to pass the entire variant (have it a named type) and even if, that code would at most have to accept two arguments
b) The main use case is to match on the variant and extract the fields, which is more brief with struct variants.