Separate typeck/trans into distinct crates.

[nikomatsakis]

This has the goal of further reducing peak memory usage and enabling more parallelism. This patch should allow trans/typeck to build in parallel. The plan is to proceed by moving as many additional passes as possible into distinct crates that lay alongside typeck/trans. Basically, the idea is that there is the rustc crate which defines the common data structures shared between passes. Individual passes then go into their own crates. Finally, the rustc_driver crate knits it all together.

cc @jakub-: One wrinkle is the diagnostics plugin. Currently, it assumes all diagnostics are defined and used within one crate in order to track what is used and what is duplicated. I had to disable this. We'll have to find an alternate strategy, but I wasn't sure what was best so decided to just disable the duplicate checking for now.

[rust-highfive]

Warning

• These commits modify unsafe code. Please review it carefully!

[nikomatsakis]

cc @alexcrichton

[bstrie]

You're a machine!

[alexcrichton]

stray file?

[alexcrichton]

@nikomatsakis out of curiosity can you check the distribution tarball size before/after this? Before we commit too much to this route I just wanna be aware if we're increasing our distribution size by many MB for each split. I suspect it won't have too too much impact, but may as well be safe!

[nikomatsakis]

@alexcrichton sure, good question. How do I check that? (some makefile target, I guess?)

I should also do some timings. I am curious to see how much (if any) this helps with make all time.

[japaric]

Heads up: I was compiling this on top of master to collect distribution size information, but it doesn't compile as it is:

src/librustc_typeck/lib.rs:68:1: 68:39 error: unused attribute, #[deny(unused_attributes)] on by default
src/librustc_typeck/lib.rs:68 #![comment = "The Rust type checker."]
                              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/librustc_typeck/lib.rs:69:1: 69:25 error: unused attribute, #[deny(unused_attributes)] on by default
src/librustc_typeck/lib.rs:69 #![license = "MIT/ASL2"]

(I didn't know that raised a warning now!)

[japaric]

Also it seems librustdoc has not been integrated yet, that crate still has references to librustc_trans::driver

[alexcrichton]

@nikomatsakis I think that make dist will put a set of tarballs in dist, or at least somewhere in the build tree

[nikomatsakis]

@japaric yeah, I guess I got trigger-happy and didn't let the make process run all the way through for the last commit. Interestingly, after fixing various minor things, I am now encountering a stack overflow (!). Not sure yet what's going on, though it looks legit -- i.e., it's just overflowing the stack. But I could be wrong.

[nikomatsakis]

(What's odd is that this occurs in the driver crate)

[nikomatsakis]

Status: I am narrowing down precisely what change causes the overflow

[nikomatsakis]

So this is rebased and fully functional.

@alexcrichton in 629b567 I upped the stack size to 32MB. This is probably bad for some reason and I suspect we could get away with less. Thoughts?

[japaric]

@nikomatsakis Compilation error after checking out this branch:

librustc_trans/back/link.rs:157:12: 157:21 error: unresolved enum variant, struct or const `FileInput`
src/librustc_trans/back/link.rs:157     if let FileInput(ref path) = *input {

[japaric]

-    if let FileInput(ref path) = *input {
+    if let Input::File(ref path) = *input {

[nikomatsakis]

@alexcrichton the distribution size is basically unaffected by this patch. I see the normal distribution as 108MB and the version on this branch as 108MB.

[nikomatsakis]

@japaric stupid rebase, thanks :)

[japaric]

@alexcrichton Can confirm that the distribution size is pretty much unaffected:

# Before
$ cave size rust
207.81 MBytes 
# After
$ cave size rust
208.00 MBytes 

Here's the size of each file

Will try to collect some timing information

[alexcrichton]

Awesome, thanks for the investigation @nikomatsakis and @japaric!

[alexcrichton]

@nikomatsakis also upping the stack size seems ok to me.

[japaric]

Timing information:

Codegen	master	This PR
Single	30:24.51 (113% CPU)	26:34.38 (131% CPU)
4 units	17:00.05 (224% CPU)	16:44.21 (263% CPU)
8 units	20:08.98 (343% CPU)	19:12.69 (314% CPU)

More detailed (per crate-ish) timings

A nice 230 seconds (12.6%) speedup for the fully optimized (no parallel codegen) case.

[nikomatsakis]

Rebased!

[nikomatsakis]

I am going to start giving this p=1 precedence because it can improve cycle time, memory usage, and (perhaps most importantly) incremental recompilation, and thus has a "waterfall" effect on later productivity.