rustc_metadata: Load metadata for indirect macro-only dependencies

[petrochenkov]

Imagine this dependency chain between crates

Executable crate -> Library crate -> Macro crate

where "Library crate" uses the macros from "Macro crate" for some code generation, but doesn't reexport them any further.

Currently, when compiling "Executable crate" we don't even load metadata for it, because why would we want to load any metadata from "Macro crate" if it already did all its code generation job when compiling "Library crate".
Right?

Wrong!
Hygiene data and spans (#68686, #68941) from "Macro crate" still may need to be decoded from "Executable crate".
So we'll have to load them properly.

Questions:

• How this will affect compile times for larger crate trees in practice? How to measure it?
  Hygiene/span encoding/decoding will necessarily slow down compilation because right now we just don't do some work that we should do, but this introduces a whole new way to slow down things. E.g. loading metadata for syn (and its dependencies) when compiling your executable if one of its library dependencies uses it.
• We are currently detecting whether a crate reexports macros from "Macro crate" or not, could we similarly detect whether a crate "reexports spans" and keep it unloaded if it doesn't?
  Or at least "reexports important spans" affecting hygiene, we can probably lose spans that only affect diagnostics.

[rust-highfive]

r? @matthewjasper

(rust_highfive has picked a reviewer for you, use r? to override)

[petrochenkov]

cc @rust-lang/compiler @ehuss @Aaron1011

[eddyb]

How this will affect compile times for larger crate trees in practice? How to measure it?

Ideally we'd just make it lazy enough for it to be cheap unless the spans are used (e.g. by decoding a mir::Body using Spans from the macro crate).

I'm curious if we have any crates on perf.r-l.o that may be affected:
@bors try @rust-timer queue

[rust-timer]

Awaiting bors try build completion

[bors]

⌛ Trying commit 487c378 with merge 341097d...

[bors]

☀️ Try build successful - checks-azure
Build commit: 341097d (341097d4377bd629984a3192d2011cb2f21cd4db)

[rust-timer]

Queued 341097d with parent d9a328a, future comparison URL.

[petrochenkov]

The performance diff looks like a noise, but I'm not sure what time is measured by perf.r-l.o.
Is it the target crate only, or the dependencies as well.

Also, why am I even asking this.
The slowdown should affect the target crate as well, even if it's measured separately.
It loads metadata from all its dependency graph, including those new macro dependencies from this PR.

[eddyb]

syn-opt is a noise generator, but the rest looks like <0.5% which seems acceptable IMO.

[petrochenkov]

Yeah, perhaps this can be just landed then, without further optimizations.

The parts that can be potentially skipped (to what extent?) are fn load and fn register_crate in librustc_metadata/creader.rs which 1) perform file search, 2) load metadata as a binary blob and 3) decode some minimal metadata about the crate root.
AFAIK, those actions were somewhat optimized in the past to speedup zero-change rebuilds.

[eddyb]

r=me if there are no concerns unrelated to perf left

[petrochenkov]

(I'll leave this pending for a few days.)

[petrochenkov]

@bors r=eddyb

[bors]

📌 Commit 487c378 has been approved by eddyb

[bors]

⌛ Testing commit 487c378 with merge 9dc8dad...

[bors]

☀️ Test successful - checks-azure
Approved by: eddyb
Pushing 9dc8dad to master...