Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Print a summary of which test suite failed #78705

Merged
merged 1 commit into from
Nov 6, 2020

Conversation

Mark-Simulacrum
Copy link
Member

@Mark-Simulacrum Mark-Simulacrum commented Nov 3, 2020

Especially on CI, where cross-compiling is common and single builder may end up
with multiple hosts and multiple targets, it can be annoying to scroll back to
the nearest start of test marker. This prints out a summary of the test suite
being run directly in compiletest.

For example, on a mir-opt failure, this would show something like this:

failures:
    [mir-opt] mir-opt/while-storage.rs

test result: FAILED. 140 passed; 1 failed; 2 ignored; 0 measured; 0 filtered out

Some tests failed in compiletest suite=mir-opt mode=mir-opt host=x86_64-unknown-linux-gnu target=x86_64-unknown-linux-gnu

Fixes #78517

@rust-highfive
Copy link
Collaborator

@Mark-Simulacrum: no appropriate reviewer found, use r? to override

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Nov 3, 2020
@Mark-Simulacrum
Copy link
Member Author

r? @jyn514 perhaps?

cc @RalfJung

@RalfJung
Copy link
Member

RalfJung commented Nov 3, 2020

Looks great!
I guess what I am most surprised by is how far away this is from the code that prints the "header" above the tests:

"Check compiletest suite={} mode={} ({} -> {})",

Do you want to keep these two in sync in terms of output? One uses HOST -> TARGET, the other host=HOST, target=TARGET.

Especially on CI, where cross-compiling is common and single builder may end up
with multiple hosts and multiple targets, it can be annoying to scroll back to
the nearest start of test marker. This prints out a summary of the test suite
being run directly in compiletest.
@Mark-Simulacrum
Copy link
Member Author

IMO the HOST -> TARGET terminology is a bit confusing and likely not immediately obvious; I don't personally care to change the (many) locations in bootstrap right now, but I also think consistency in this case is not too important.

I think this could be in bootstrap but -- as you noted on the issue -- it'd be annoying to pipe the relevant data in, and here we immediately have it available.

@jyn514 jyn514 added A-contributor-roadblock Area: Makes things more difficult for new contributors to rust itself T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Nov 4, 2020
@jyn514
Copy link
Member

jyn514 commented Nov 4, 2020

Some tests failed in compiletest suite=mir-opt mode=mir-opt

Will the suite and mode ever be different? r=me if so, but otherwise I don't think we need to print both.

@jyn514 jyn514 added A-testsuite Area: The testsuite used to check the correctness of rustc C-enhancement Category: An issue proposing an enhancement or a PR with one. labels Nov 4, 2020
@Mark-Simulacrum
Copy link
Member Author

I don't have the time to do a detailed investigation, but I suspect the answer is yes (or, at least, that this is plausible). The two are tracked and passed as separate things in bootstrap as well.

@jyn514
Copy link
Member

jyn514 commented Nov 4, 2020

Ok, they can be different:

host_test!(UiFullDeps { path: "src/test/ui-fulldeps", mode: "ui", suite: "ui-fulldeps" });
--
    mode: "run-make",
    suite: "run-make-fulldeps"
--
                mode: "js-doc-test",
                suite: "rustdoc-js",
--
            mode: "ui",
            suite: "rustdoc-ui",
--

@bors r+

@bors
Copy link
Contributor

bors commented Nov 4, 2020

📌 Commit f289a87 has been approved by jyn514

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 4, 2020
@bors
Copy link
Contributor

bors commented Nov 4, 2020

💡 This pull request was already approved, no need to approve it again.

@bors
Copy link
Contributor

bors commented Nov 4, 2020

📌 Commit f289a87 has been approved by jyn514

bors added a commit to rust-lang-ci/rust that referenced this pull request Nov 6, 2020
Rollup of 15 pull requests

Successful merges:

 - rust-lang#74979 (`#![deny(unsafe_op_in_unsafe_fn)]` in sys/hermit)
 - rust-lang#78006 (Use Intra-doc links for std::io::buffered)
 - rust-lang#78167 (Fix unreachable sub-branch detection in or-patterns)
 - rust-lang#78514 (Allow using 1/2/3/4 for `x.py setup` options)
 - rust-lang#78538 (BTreeMap: document a curious assumption in test cases)
 - rust-lang#78559 (Add LLVM upgrades from 7 to 10 to RELEASES.md)
 - rust-lang#78666 (Fix shellcheck error)
 - rust-lang#78705 (Print a summary of which test suite failed)
 - rust-lang#78726 (Add link to rust website)
 - rust-lang#78730 (Expand explanation of reverse_bits)
 - rust-lang#78760 (`deny(invalid_codeblock_attributes)` for rustc_error_codes)
 - rust-lang#78771 (inliner: Copy unevaluated constants only after successful inlining)
 - rust-lang#78794 (rustc_expand: use collect_bang helper instead of manual reimplementation)
 - rust-lang#78795 (The renumber pass is long gone)
 - rust-lang#78798 (Fixing Spelling Typos)

Failed merges:

r? `@ghost`
`@rustbot` modify labels: rollup
@bors bors merged commit 3f5723c into rust-lang:master Nov 6, 2020
@rustbot rustbot added this to the 1.49.0 milestone Nov 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-contributor-roadblock Area: Makes things more difficult for new contributors to rust itself A-testsuite Area: The testsuite used to check the correctness of rustc C-enhancement Category: An issue proposing an enhancement or a PR with one. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

compiletest: show affected architecture in error summary
6 participants