mk: Bootstrap from stable instead of snapshots #32942

alexcrichton · 2016-04-13T20:47:58Z

This commit removes all infrastructure from the repository for our so-called
snapshots to instead bootstrap the compiler from stable releases. Bootstrapping
from a previously stable release is a long-desired feature of distros because
they're not fans of downloading binary stage0 blobs from us. Additionally, this
makes our own CI easier as we can decommission all of the snapshot builders and
start having a regular cadence to when we update the stage0 compiler.

A new src/etc/get-stage0.py script was added which shares some code with
src/bootstrap/bootstrap.py to read a new file, src/stage0.txt, which lists
the current stage0 compiler as well as cargo that we bootstrap from. This script
will download the relevant rustc package an unpack it into $target/stage0 as
we do today.

One problem of bootstrapping from stable releases is that we're not able to
compile unstable code (e.g. all the #![feature] directives in libcore/libstd).
To overcome this we employ two strategies:

The bootstrap key of the previous compiler is hardcoded into src/stage0.txt
(enabled as a result of mk: Hardcode the bootstrap key for each release #32731) and exported by the build system. This enables
nightly features in the compiler we download.
The standard library and compiler are pinned to a specific stage0, which
doesn't change, so we're guaranteed that we'll continue compiling as we start
from a known fixed source.

The process for making a release will also need to be tweaked now to continue to
cadence of bootstrapping from the previous release. This process looks like:

Merge beta to stable
Produce a new stable compiler.
Change master to bootstrap from this new stable compiler.
Merge master to beta
Produce a new beta compiler
Change master to bootstrap from this new beta compiler.

Step 3 above should involve very few changes as master was previously
bootstrapping from beta which is the same as stable at that point in time.
Step 6, however, is where we benefit from removing lots of #[cfg(stage0)] and
get to use new features. This also shouldn't slow the release too much as steps
1-5 requires little work other than waiting and step 6 just needs to happen at
some point during a release cycle, it's not time sensitive.

Closes #29555
Closes #29557

rust-highfive · 2016-04-13T20:48:03Z

r? @aturon

(rust_highfive has picked a reviewer for you, use r? to override)

alexcrichton · 2016-04-13T20:52:10Z

r? @brson

This commit moves us to bootstrapping from the recently-produced 1.9 beta compiler as a proof-of-concept for both the makefiles and rustbuild.

cc @dhuseby, @mneumann, @semarie -- this will unfortunately break BSD bootstraps for the near future. We don't have stable releases for platforms like OpenBSD, Bitrig, or DragonFly (we do for FreeBSD/NetBSD now), but we'd have to hook up some more automation to get this working.

In the interim we can perhaps produce releases after the fact for these platforms, but it would be best if we could figure out how to cross-compile these platforms from Linux (like FreeBSD and NetBSD are doing). That way we could hook everything up to our automation, make sure releases/nightlies are made, everything keeps building, etc.

brson · 2016-04-13T23:41:02Z

r=me but we should probably hear from the *BSD maintainers before going forward.

dhuseby · 2016-04-14T01:09:04Z

@alexcrichton so we should all focus on getting cross-compiling from linux to work? that's trivial for netbsd and probably works for freebsd, but openbsd is likely much harder due to the use of older/custom versions of the gnu toolchain. I know that bitrig will be very difficult as they rely solely on clang and libc++. Convincing Linux to only use that toolchain and libraries is not trivial. The last time I figure it out it looked like this:

compile:

clang++ -std=c++11 -nostdlib -mstackrealign -g -I/usr/include/c++/v1/ -I/usr/include/libcxxabi -I/usr/lib/llvm-3.4/include  -DNDEBUG -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -g -O2 -fomit-frame-pointer -fvisibility-inlines-hidden -fno-exceptions -fPIC -Woverloaded-virtual -Wcast-qual -c test2.cc

link:

/usr/bin/ld -z relro --hash-style=gnu --build-id --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o test2 /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crt1.o /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crti.o /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/crtbegin.o -L/usr/lib/llvm-3.4/lib -L/usr/lib/x86_64-linux-gnu/ -L/lib64 -L/lib -L/usr/lib test2.o -lc++ -lc++abi -lunwind -lc -lpthread -lffi -ltinfo -ldl -lm /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/crtend.o /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crtn.o

ldd:

        linux-vdso.so.1 =>  (0x00007fff5d7fe000)
        libc++.so.1 => /usr/lib/x86_64-linux-gnu/libc++.so.1 (0x00007fac64095000)
        libc++abi.so.1 => /usr/lib/x86_64-linux-gnu/libc++abi.so.1 (0x00007fac63e46000)
        libunwind.so.8 => /usr/lib/x86_64-linux-gnu/libunwind.so.8 (0x00007fac63c2a000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fac63864000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fac63646000)
        libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007fac6343d000)
        libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007fac63214000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fac63010000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fac62d09000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fac62b01000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fac643ad000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fac628df000)

dhuseby · 2016-04-14T01:09:35Z

@alexcrichton I'm all for figuring this out as the CI will be much more uniform and cleaner.

alexcrichton · 2016-04-14T06:51:21Z

@alexcrichton so we should all focus on getting cross-compiling from linux to work?

For now, yeah, that's our best option for keeping these platforms going. We can very easily set up and manage bots to cross-compile to each BSD, but running dedicated BSD hardware is a much bigger headache for us we aren't willing to stomach just yet.

that's trivial for netbsd and probably works for freebsd

Indeed! We've got nightlies/betas for both of them :)

but openbsd is likely much harder due to the use of older/custom versions of the gnu toolchain.

Yeah this is kinda unfortunate :(. There may not be any way to cross from Linux in the long run.

In general this doesn't necessarily mean "game over" for the BSDs, it may just mean that we periodically upload a new beta for each BSD. We probably don't want to push stable releases just yet unless we've got "more confidence" in them, but we could also perhaps do that as well!

In essence this'll just mean that new BSD "snapshots" will need to happen on a 6-week cadence

mneumann · 2016-04-14T08:25:37Z

@alexcrichton I don't see any problems with this change for DragonFly. We are already using FreeBSD's ports infrastructure for building rustc (with some minor patches) and I think the FreeBSD ports maintainers will do all the work for us :)

alexcrichton · 2016-04-14T17:50:53Z

Thanks for the update @mneumann! Ok, @dhuseby so you think it's ok to merge this and we can figure out perhaps making beta releases for these other platforms afterwards?

bors · 2016-04-15T01:21:26Z

☔ The latest upstream changes (presumably #32960) made this pull request unmergeable. Please resolve the merge conflicts.

semarie · 2016-04-15T03:41:41Z

@alexcrichton if I correctly understand, for openbsd it will be required to generate a -beta release each 6 weeks (if not crosscompile possibility) ? for me, it won't be a big problem: I could generate it for upload.

does snap3 buildbots will be converted to beta buildbots ?

bors · 2016-04-15T13:19:20Z

☔ The latest upstream changes (presumably #32895) made this pull request unmergeable. Please resolve the merge conflicts.

alexcrichton · 2016-04-15T16:37:47Z

@semarie yeah if we want to keep nightly OpenBSD compiling we'll basically need to simulate a beta release for these platforms whenever a beta is built (this includes beta updates as well if we switch to bootstrapping from them).

We'll be retiring the snap3 buildbots and we do indeed have a suite of beta buildbots. We could add an OpenBSD one there, but I'm not sure the snap3 bots have ever produced an OpenBSD snapshot successfully so we may not have much luck with the beta ones either :(

(this is where cross-compiling to OpenBSD is the easiest route for us)

bors · 2016-04-19T04:10:17Z

☔ The latest upstream changes (presumably #32755) made this pull request unmergeable. Please resolve the merge conflicts.

brson · 2016-04-19T17:27:56Z

@bors r+

bors · 2016-04-19T17:27:57Z

📌 Commit d291511 has been approved by brson

alexcrichton · 2016-04-19T17:30:41Z

@bors: r-

Gah just realized the travis error is probably legit.

alexcrichton · 2016-04-19T17:33:48Z

@bors: r=brson 7a3ac97

bors · 2016-04-19T17:51:47Z

⌛ Testing commit 7a3ac97 with merge 04e9d72...

bors · 2016-04-19T17:53:32Z

💔 Test failed - auto-linux-64-x-freebsd

This commit removes all infrastructure from the repository for our so-called snapshots to instead bootstrap the compiler from stable releases. Bootstrapping from a previously stable release is a long-desired feature of distros because they're not fans of downloading binary stage0 blobs from us. Additionally, this makes our own CI easier as we can decommission all of the snapshot builders and start having a regular cadence to when we update the stage0 compiler. A new `src/etc/get-stage0.py` script was added which shares some code with `src/bootstrap/bootstrap.py` to read a new file, `src/stage0.txt`, which lists the current stage0 compiler as well as cargo that we bootstrap from. This script will download the relevant `rustc` package an unpack it into `$target/stage0` as we do today. One problem of bootstrapping from stable releases is that we're not able to compile unstable code (e.g. all the `#![feature]` directives in libcore/libstd). To overcome this we employ two strategies: * The bootstrap key of the previous compiler is hardcoded into `src/stage0.txt` (enabled as a result of rust-lang#32731) and exported by the build system. This enables nightly features in the compiler we download. * The standard library and compiler are pinned to a specific stage0, which doesn't change, so we're guaranteed that we'll continue compiling as we start from a known fixed source. The process for making a release will also need to be tweaked now to continue to cadence of bootstrapping from the previous release. This process looks like: 1. Merge `beta` to `stable` 2. Produce a new stable compiler. 3. Change `master` to bootstrap from this new stable compiler. 4. Merge `master` to `beta` 5. Produce a new beta compiler 6. Change `master` to bootstrap from this new beta compiler. Step 3 above should involve very few changes as `master` was previously bootstrapping from `beta` which is the same as `stable` at that point in time. Step 6, however, is where we benefit from removing lots of `#[cfg(stage0)]` and get to use new features. This also shouldn't slow the release too much as steps 1-5 requires little work other than waiting and step 6 just needs to happen at some point during a release cycle, it's not time sensitive. Closes rust-lang#29555 Closes rust-lang#29557

alexcrichton · 2016-04-19T17:57:04Z

@bors: r=brson 02538d4

bors · 2016-04-19T19:23:58Z

⌛ Testing commit 02538d4 with merge 31ffdf4...

alexcrichton · 2016-04-19T20:17:07Z

@bors: retry force

bors · 2016-04-19T22:06:56Z

⌛ Testing commit 02538d4 with merge cc0dada...

bors · 2016-04-19T23:54:16Z

💔 Test failed - auto-win-msvc-32-opt

alexcrichton · 2016-04-20T01:11:42Z

@bors: retry

On Tuesday, April 19, 2016, bors notifications@github.com wrote:

[image: 💔] Test failed - auto-win-msvc-32-opt
http://buildbot.rust-lang.org/builders/auto-win-msvc-32-opt/builds/3067

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#32942 (comment)

bors · 2016-04-20T08:16:56Z

⌛ Testing commit 02538d4 with merge 9bba290...

mk: Bootstrap from stable instead of snapshots This commit removes all infrastructure from the repository for our so-called snapshots to instead bootstrap the compiler from stable releases. Bootstrapping from a previously stable release is a long-desired feature of distros because they're not fans of downloading binary stage0 blobs from us. Additionally, this makes our own CI easier as we can decommission all of the snapshot builders and start having a regular cadence to when we update the stage0 compiler. A new `src/etc/get-stage0.py` script was added which shares some code with `src/bootstrap/bootstrap.py` to read a new file, `src/stage0.txt`, which lists the current stage0 compiler as well as cargo that we bootstrap from. This script will download the relevant `rustc` package an unpack it into `$target/stage0` as we do today. One problem of bootstrapping from stable releases is that we're not able to compile unstable code (e.g. all the `#![feature]` directives in libcore/libstd). To overcome this we employ two strategies: * The bootstrap key of the previous compiler is hardcoded into `src/stage0.txt` (enabled as a result of #32731) and exported by the build system. This enables nightly features in the compiler we download. * The standard library and compiler are pinned to a specific stage0, which doesn't change, so we're guaranteed that we'll continue compiling as we start from a known fixed source. The process for making a release will also need to be tweaked now to continue to cadence of bootstrapping from the previous release. This process looks like: 1. Merge `beta` to `stable` 2. Produce a new stable compiler. 3. Change `master` to bootstrap from this new stable compiler. 4. Merge `master` to `beta` 5. Produce a new beta compiler 6. Change `master` to bootstrap from this new beta compiler. Step 3 above should involve very few changes as `master` was previously bootstrapping from `beta` which is the same as `stable` at that point in time. Step 6, however, is where we benefit from removing lots of `#[cfg(stage0)]` and get to use new features. This also shouldn't slow the release too much as steps 1-5 requires little work other than waiting and step 6 just needs to happen at some point during a release cycle, it's not time sensitive. Closes #29555 Closes #29557

bors · 2016-04-20T10:44:54Z

Unsupport wget wget support was removed in rust-lang#32942 (search for wget in diff), but configure wasn't updated. wget support was introduced in rust-lang#7498 for Windows, but we now use PowerShell on Windows.

rust-highfive assigned aturon Apr 13, 2016

alexcrichton force-pushed the bootstrap-from-previous branch from a894fc0 to dc4b0b1 Compare April 13, 2016 20:49

rust-highfive assigned brson and unassigned aturon Apr 13, 2016

alexcrichton force-pushed the bootstrap-from-previous branch from dc4b0b1 to 3244b75 Compare April 14, 2016 17:49

alexcrichton force-pushed the bootstrap-from-previous branch from 3244b75 to cfc260e Compare April 15, 2016 01:22

alexcrichton force-pushed the bootstrap-from-previous branch 4 times, most recently from 39e4944 to baccb8f Compare April 15, 2016 20:46

alexcrichton force-pushed the bootstrap-from-previous branch 2 times, most recently from c757d35 to d291511 Compare April 19, 2016 15:41

alexcrichton force-pushed the bootstrap-from-previous branch from d291511 to 7a3ac97 Compare April 19, 2016 17:33

alexcrichton force-pushed the bootstrap-from-previous branch from 7a3ac97 to 02538d4 Compare April 19, 2016 17:56

bors mentioned this pull request Apr 19, 2016

debuginfo: Don't emit DW_LANG_RUST unless it's explicitly demanded #33097

Closed

alexcrichton mentioned this pull request Apr 19, 2016

Python code cleanup, first work toward CPython 3 compatibility #33036

Closed

bors merged commit 02538d4 into rust-lang:master Apr 20, 2016

bors mentioned this pull request Apr 20, 2016

adding Dragonfly x86_64 snapshot 235d774 #32864

Closed

sanxiyn mentioned this pull request Apr 26, 2016

Warn unused trait imports, rebased #33091

Merged

alexcrichton deleted the bootstrap-from-previous branch May 2, 2016 18:44

sanxiyn mentioned this pull request Jun 3, 2016

Unsupport wget #34062

Merged

cuviper mentioned this pull request Jul 1, 2016

Release notes for 1.10.0 #34591

Merged

japaric mentioned this pull request Jul 19, 2016

How to compile rust in a musl libc based linux system #31322

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mk: Bootstrap from stable instead of snapshots #32942

mk: Bootstrap from stable instead of snapshots #32942

alexcrichton commented Apr 13, 2016

rust-highfive commented Apr 13, 2016

alexcrichton commented Apr 13, 2016

brson commented Apr 13, 2016

dhuseby commented Apr 14, 2016

dhuseby commented Apr 14, 2016

alexcrichton commented Apr 14, 2016

mneumann commented Apr 14, 2016

alexcrichton commented Apr 14, 2016

bors commented Apr 15, 2016

semarie commented Apr 15, 2016

bors commented Apr 15, 2016

alexcrichton commented Apr 15, 2016

bors commented Apr 19, 2016

brson commented Apr 19, 2016

bors commented Apr 19, 2016

alexcrichton commented Apr 19, 2016

alexcrichton commented Apr 19, 2016

bors commented Apr 19, 2016

bors commented Apr 19, 2016

alexcrichton commented Apr 19, 2016

bors commented Apr 19, 2016

alexcrichton commented Apr 19, 2016

bors commented Apr 19, 2016

bors commented Apr 19, 2016

alexcrichton commented Apr 20, 2016

bors commented Apr 20, 2016

bors commented Apr 20, 2016

mk: Bootstrap from stable instead of snapshots #32942

mk: Bootstrap from stable instead of snapshots #32942

Conversation

alexcrichton commented Apr 13, 2016

rust-highfive commented Apr 13, 2016

alexcrichton commented Apr 13, 2016

brson commented Apr 13, 2016

dhuseby commented Apr 14, 2016

dhuseby commented Apr 14, 2016

alexcrichton commented Apr 14, 2016

mneumann commented Apr 14, 2016

alexcrichton commented Apr 14, 2016

bors commented Apr 15, 2016

semarie commented Apr 15, 2016

bors commented Apr 15, 2016

alexcrichton commented Apr 15, 2016

bors commented Apr 19, 2016

brson commented Apr 19, 2016

bors commented Apr 19, 2016

alexcrichton commented Apr 19, 2016

alexcrichton commented Apr 19, 2016

bors commented Apr 19, 2016

bors commented Apr 19, 2016

alexcrichton commented Apr 19, 2016

bors commented Apr 19, 2016

alexcrichton commented Apr 19, 2016

bors commented Apr 19, 2016

bors commented Apr 19, 2016

alexcrichton commented Apr 20, 2016

bors commented Apr 20, 2016

bors commented Apr 20, 2016