Stop keeping metadata in memory before writing it to disk

[cjgillot]

rustc_metadata encodes all the crate information into an in-memory buffer in EncodedMetadata. This in-memory buffer is then passed around to be written to disk, as an rmeta file in rustc_interface::encode_and_write_metadata and/or in an rlib/dylib by codegen. Codegen wraps metadata into an object file using the object crate to be nice to linkers that will manipulate the rlib/dylib.

Keeping metadata in-memory increases the amount of memory used by rustc significantly. We want to investigate saving this memory by writing metadata to disk early, reading it back if required.

Instructions:

• Move rustc_codegen_ssa::back::link::emit_metadata and rustc_codegen_ssa::METADATA_FILENAME to new module rustc_metadata::fs.
• Move rustc_interface::passes::encode_and_write_metadata and rustc_metadata::util::non_durable_rename to rustc_metadata::fs.
• Write metadata into the temporary file in encode_and_write_metadata even if !need_metadata_file.
• Use a rustc_serialize::opaque::FileEncoder in rustc_metadata::rmeta::encoder instead of an opaque::Encoder, it should encode directly to the temporary file, and re-read this temporary file to build the EncodedMetadata.
• Change EncodedMetadata to hold a Mmap of the on-disk metadata instead of a Vec. This Mmap can be of the temporary file, or of the proper renamed output file. In the former case, EncodedMetadata should carry the MaybeTempDir to avoid deleting the temporary directory while accessing the Mmap.

Extra: the current implementations of create_rmeta_file and create_compressed_metadata_file in rustc_codegen_ssa::back::metadata buffer everything to a vector before writing it all at once. This could be refactored to write directly to the file using a BufWriter.

Please reach out on Zulip if you have any questions. Preferably on a public stream so experts on codegen can weigh in when necessary.

[m-ysk]

@rustbot claim

[bjorn3]

Write metadata into the temporary file in encode_and_write_metadata even if !need_metadata_file.

Preferably only do this if either need_metadata_file or should_codegen().