Move toolstate checking into bootstrap #66681

Mark-Simulacrum · 2019-11-23T23:22:48Z

This intends no functional changes, merely translates the spread of shell/python into Rust.

One problematic area that I'd like to avoid but wasn't quite able to figure out how is the master branch script which is still in bash/python -- I cared less about that since it is orthogonal to the actual checking that we're doing, though as-is we're duplicating some code across Rust and that script.

r? @kennytm or maybe @pietroalbini

Mark-Simulacrum · 2019-11-23T23:24:34Z

This is preparation for #65000 -- but not the changes that the FCP there is about; that will be done in a separate, future PR.

rust-highfive · 2019-11-24T01:24:47Z

The job x86_64-gnu-tools of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

2019-11-23T23:26:09.6705837Z ##[command]git remote add origin https://github.com/rust-lang/rust
2019-11-23T23:26:09.6921905Z ##[command]git config gc.auto 0
2019-11-23T23:26:09.6994396Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2019-11-23T23:26:10.6605374Z ##[command]git config --get-all http.proxy
2019-11-23T23:26:10.6608286Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/66681/merge:refs/remotes/pull/66681/merge
---
2019-11-23T23:30:44.5173977Z  ---> 6de740cc5443
2019-11-23T23:30:44.5213330Z Successfully built 6de740cc5443
2019-11-23T23:30:44.6441849Z Successfully tagged rust-ci:latest
2019-11-23T23:30:44.6746881Z Built container sha256:6de740cc54439184d46c95f3338940ceb4997f9599afd79cdde112c1444c6be1
2019-11-23T23:30:44.6764760Z Uploading finished image to https://rust-lang-ci-sccache2.s3.amazonaws.com/docker/480929bc3bc019c75d577647f444b4de6781d09e6c536dbea2d03afb25b3d85ad948e63811757d340a630b00995141e1b0ee59f2e576e5c502c9156bf4ec6fc2
2019-11-23T23:31:13.6028864Z upload failed: - to s3://rust-lang-ci-sccache2/docker/480929bc3bc019c75d577647f444b4de6781d09e6c536dbea2d03afb25b3d85ad948e63811757d340a630b00995141e1b0ee59f2e576e5c502c9156bf4ec6fc2 An error occurred (InvalidAccessKeyId) when calling the CreateMultipartUpload operation: The AWS Access Key Id you provided does not exist in our records.
2019-11-23T23:31:14.6805829Z [CI_JOB_NAME=x86_64-gnu-tools]
2019-11-23T23:31:14.6848138Z == clock drift check ==
2019-11-23T23:31:14.6857200Z   local time: Sat Nov 23 23:31:14 UTC 2019
2019-11-23T23:31:14.9748892Z   network time: Sat, 23 Nov 2019 23:31:14 GMT

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

Mark-Simulacrum · 2019-11-24T20:28:55Z

(CI failure is expected and due to the last commit which tests that we're still verifying submodules)

bors · 2019-11-25T22:48:41Z

☔ The latest upstream changes (presumably #66739) made this pull request unmergeable. Please resolve the merge conflicts.

Mark-Simulacrum · 2019-12-02T13:39:15Z

r? @pietroalbini

Mark-Simulacrum · 2019-12-02T13:39:33Z

(If we're down with the approach the rebase and such here should be mostly painless)

pietroalbini

One problematic area that I'd like to avoid but wasn't quite able to figure out how is the master branch script which is still in bash/python -- I cared less about that since it is orthogonal to the actual checking that we're doing, though as-is we're duplicating some code across Rust and that script.

I don't think it's that bad if we initialize the full CI environment in the master builder (it's like a couple of minutes?). Then we can just do ./x.py test publish-toolstate (switching to run once my GHA PR lands).

src/bootstrap/toolstate.rs

pietroalbini · 2019-12-09T10:20:43Z

src/bootstrap/toolstate.rs

+///
+///       * See <https://help.github.com/articles/about-commit-email-addresses/>
+///           if a private email by GitHub is wanted.
+fn commit_toolstate_change(


Can we switch to use deploy keys instead? Dunno if you want to do this in this PR or another, but it's something I think we should do eventually. The code to push with a deploy key is here.

I'd rather do this in a separate PR, seems cleaner.

src/bootstrap/toolstate.rs

Mark-Simulacrum · 2019-12-09T16:46:49Z

I don't think it's that bad if we initialize the full CI environment in the master builder (it's like a couple of minutes?). Then we can just do ./x.py test publish-toolstate (switching to run once my GHA PR lands).

Yeah, this seems reasonable. I would prefer that we land this and then do that in a followup PR -- I can work on it pretty quickly (mostly because I worry about changing too much in one PR, and the existing duplication is pretty minimal).

I believe I've resolve the other review comments you left.

pietroalbini · 2019-12-10T13:34:19Z

@bors r+ rollup=never

Thanks!

bors · 2019-12-10T13:34:21Z

📌 Commit 3e44c9c98b080561104dec7a797fe1901de43704 has been approved by pietroalbini

bors · 2019-12-10T13:36:55Z

⌛ Testing commit 3e44c9c98b080561104dec7a797fe1901de43704 with merge a51db9dafcc9ea5695bbde6cd14d0cdc20221b9f...

rust-highfive · 2019-12-10T13:44:36Z

The job x86_64-gnu-tools of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

bors · 2019-12-10T13:44:42Z

💔 Test failed - checks-azure

This is not yet actually used by CI, but implements the logic for checking that tools are properly building on beta/stable and during beta cutoff week. This attempts to mirror the checking functionality in src/ci/docker/x86_64-gnu-tools/checktools.sh, and called scripts. It does not attempt to run the relevant steps (that functionality was originally desired to be moved into bootstrap as well, but doing so proved more difficult than expected). This is intended as a way to centralize and make clearer the logic involved in toolstate checking. In particular, the previous logic was spread across numerous python and shell scripts in such a way that made interpretation quite difficult.

Mark-Simulacrum · 2019-12-10T14:30:39Z

@bors r=pietroalbini

bors · 2019-12-10T14:30:41Z

📌 Commit 97d9364 has been approved by pietroalbini

resolved

bors · 2019-12-10T16:50:52Z

⌛ Testing commit 97d9364 with merge 7dbfb0a...

@kennytm

…etroalbini Move toolstate checking into bootstrap This intends no functional changes, merely translates the spread of shell/python into Rust. One problematic area that I'd like to avoid but wasn't quite able to figure out how is the master branch script which is still in bash/python -- I cared less about that since it is orthogonal to the actual checking that we're doing, though as-is we're duplicating some code across Rust and that script. r? @kennytm or maybe @pietroalbini

bors · 2019-12-10T19:59:48Z

☀️ Test successful - checks-azure
Approved by: pietroalbini
Pushing 7dbfb0a to master...

rust-highfive · 2019-12-10T20:01:20Z

Your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

rust-highfive assigned kennytm Nov 23, 2019

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Nov 23, 2019

kennytm added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 24, 2019

Mark-Simulacrum added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Nov 24, 2019

rust-highfive assigned pietroalbini and unassigned kennytm Dec 2, 2019

pietroalbini previously requested changes Dec 9, 2019

View reviewed changes

Mark-Simulacrum force-pushed the toolstate-into-bootstrap branch from c83a2fd to 3e44c9c Compare December 9, 2019 16:42

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 10, 2019

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Dec 10, 2019

Mark-Simulacrum added 2 commits December 10, 2019 09:26

Move CI to new builtin check-toolstate

97d9364

Mark-Simulacrum force-pushed the toolstate-into-bootstrap branch from 3e44c9c to 97d9364 Compare December 10, 2019 14:30

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 10, 2019

bors added the merged-by-bors This PR was explicitly merged by bors. label Dec 10, 2019

bors merged commit 97d9364 into rust-lang:master Dec 10, 2019

bors mentioned this pull request Dec 10, 2019

Stop explicitly depending on python 2 #66605

Merged

ehuss mentioned this pull request Mar 2, 2020

Toolstate: Don't block beta week on already broken tools. #69624

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move toolstate checking into bootstrap #66681

Move toolstate checking into bootstrap #66681

Mark-Simulacrum commented Nov 23, 2019

Mark-Simulacrum commented Nov 23, 2019

rust-highfive commented Nov 24, 2019

Mark-Simulacrum commented Nov 24, 2019

bors commented Nov 25, 2019

Mark-Simulacrum commented Dec 2, 2019

Mark-Simulacrum commented Dec 2, 2019

pietroalbini left a comment

pietroalbini Dec 9, 2019

Mark-Simulacrum Dec 9, 2019

Mark-Simulacrum commented Dec 9, 2019

pietroalbini commented Dec 10, 2019

bors commented Dec 10, 2019

bors commented Dec 10, 2019

rust-highfive commented Dec 10, 2019

bors commented Dec 10, 2019

Mark-Simulacrum commented Dec 10, 2019

bors commented Dec 10, 2019

bors commented Dec 10, 2019

bors commented Dec 10, 2019

rust-highfive commented Dec 10, 2019

Move toolstate checking into bootstrap #66681

Move toolstate checking into bootstrap #66681

Conversation

Mark-Simulacrum commented Nov 23, 2019

Mark-Simulacrum commented Nov 23, 2019

rust-highfive commented Nov 24, 2019

Mark-Simulacrum commented Nov 24, 2019

bors commented Nov 25, 2019

Mark-Simulacrum commented Dec 2, 2019

Mark-Simulacrum commented Dec 2, 2019

pietroalbini left a comment

Choose a reason for hiding this comment

pietroalbini Dec 9, 2019

Choose a reason for hiding this comment

Mark-Simulacrum Dec 9, 2019

Choose a reason for hiding this comment

Mark-Simulacrum commented Dec 9, 2019

pietroalbini commented Dec 10, 2019

bors commented Dec 10, 2019

bors commented Dec 10, 2019

rust-highfive commented Dec 10, 2019

bors commented Dec 10, 2019

Mark-Simulacrum commented Dec 10, 2019

bors commented Dec 10, 2019

bors commented Dec 10, 2019

bors commented Dec 10, 2019

rust-highfive commented Dec 10, 2019