Make AST->HIR lowering incremental

[cjgillot]

Implementation of MCP rust-lang/compiler-team#452

AST->HIR lowering is currently done ahead of the construction of the incremental compilation engine. This issue proposes to make lowering incremental by introducing a query lower_to_hir(def: LocalDefId) -> Option<&hir::OwnerInfo<'_>> which lowers the AST item corresponding to definition def into the corresponding HIR and some associated information.

The objective is to eventually skip lowering some parts of the AST, and to allow progress towards making resolution and expansion incremental.

The proposed design is as follows:

• remove global analyses from the current lowering pass, and pull them back to resolution stage;
• attach lowering information to definitions instead of having multiple maps inside hir::Crate, effectively making hir::Crate an IndexVec<LocalDefId, Option<OwnerInfo<'_>>>;
• extract the required resolution outputs from the resolver and keep them immutable;
• allow to create new definitions from inside the query system (*);
• index the AST by LocalDefId, using an IndexVec<LocalDefId, AstOwner> where AstOwner is an enum holding OwningRef<Lrc<Crate>, ItemLike> for ItemLike different AST item types;
• perform HIR indexing per-owner;
• make lowering incremental.

One of the most subtle changes is (*), which allows to create new definitions from inside queries. It is made safe using an eval-always query which forces re-execution of its caller, and effectively enforces the side-effects creating the definition. The first time definitions are iterated over (for instance for metadata output), a read-lock to the definitions is leaked, which seals the list of definitions.

This PR is blocked on:

• Lower only one HIR owner at a time #87234;
• Treat macros as HIR items #88019;
• Gather module items after lowering. #88703;
• Rework HIR API to make invocations of the hir_crate query harder. #88880;
• Index and hash HIR as part of lowering #89124;
• Reduce use of LocalDefId <-> HirId maps #90146;
• Store hir_id_to_def_id in OwnerNodes. #91112;
• Remove LocalDefId <-> HirId global maps #89278;
• Decouple rustc_resolve and rustc_ast_lowering #90451;
• Make lowering pull-based #90204;
• Make lowering a query #95573.

The query scheme should basically look like this:

• the untracked ast is used to compute lower_to_hir;
• lower_to_hir yields hir_attrs, hir_owner_nodes and hir_owner_parent;
• hir_owner_nodes yields hir_owner;
• hir_module_items uses hir_owner_nodes to gather all the items inside a module;
• hir_crate uses hir_owner_nodes to gather all the items inside the crate.

r? @michaelwoerister as requested on Zulip

[rust-highfive]

Some changes occurred in src/tools/clippy.

cc @rust-lang/clippy

[cjgillot]

@bors try @rust-timer queue

[rust-timer]

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

[bors]

⌛ Trying commit aa6204b46dcb5a45bc915430bfe05975d4ae38ea with merge f5e4230e3edf7c9689e3a4b9fde9426a3f7630ff...

[bors]

☀️ Try build successful - checks-actions
Build commit: f5e4230e3edf7c9689e3a4b9fde9426a3f7630ff (f5e4230e3edf7c9689e3a4b9fde9426a3f7630ff)

[rust-timer]

Queued f5e4230e3edf7c9689e3a4b9fde9426a3f7630ff with parent 59ce765, future comparison URL.

[rust-timer]

Finished benchmarking try commit (f5e4230e3edf7c9689e3a4b9fde9426a3f7630ff): comparison url.

Summary: This change led to very large relevant mixed results 🤷 in compiler performance.

• Large improvement in instruction counts (up to -4.9% on full builds of cranelift-codegen)
• Very large regression in instruction counts (up to 26.5% on incr-unchanged builds of match-stress-enum)

If you disagree with this performance assessment, please file an issue in rust-lang/rustc-perf.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR led to changes in compiler perf.

Next Steps: If you can justify the regressions found in this perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: +S-waiting-on-review -S-waiting-on-perf +perf-regression

[cjgillot]

This PR features a sizeable perf regression, up to 11%. I believe this is entirely due to query overhead: this PR requires one extra query invocation for each LocalDefId in the crate. I split #95573 which makes lowering a query, but does not separate lowering each LocalDefId.

[bors]

☔ The latest upstream changes (presumably #95662) made this pull request unmergeable. Please resolve the merge conflicts.

[michaelwoerister]

I'm setting this PR to blocked for now. #95573 looks promising!

[bors]

☔ The latest upstream changes (presumably #91557) made this pull request unmergeable. Please resolve the merge conflicts.

[bors]

☔ The latest upstream changes (presumably #93803) made this pull request unmergeable. Please resolve the merge conflicts.

[JohnCSimon]

@cjgillot
Ping from triage: I'm closing this due to inactivity, Please reopen when you are ready to continue with this.
Note: if you do please open the PR BEFORE you push to it, else you won't be able to reopen - this is a quirk of github.
Thanks for your contribution.

@rustbot label: +S-inactive