Summarize output result of all test binaries run

[davemilter]

For example if run cargo test for bindgen,
you got such output:

running 16 tests
**many lines**
test result: ok. 16 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
running 305 tests
**many lines**
test result: ok. 305 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
running 10 tests
**many lines**
test result: ok. 2 passed; 0 failed; 8 ignored; 0 measured; 0 filtered out

Normally you run it in window that not in fullscreen mode (common scenario that this window
belong to editor/ide and you have your code on the same screen at the same time),

so after all you see only last results summary, ordinary doc tests summary result, which are not important at now, it would be nice to have to have option to print
summary result at the end, like this:

Summary:
                 passed | failed |  ignored | measured
unit tests        x     |    x    |       x       |      x
func tests       x     |    x    |       x       |      x
doc tests        x     |    x    |       x       |      x

[alexcrichton]

Thanks for the report! This is something we've long wanted to improve but unfortunately haven't had a chance to design/implement yet. Most of this output comes from libtest in upstream rust-lang/rust, and this'd likely fall into the category of "custom test frameworks" as well.

[davemilter]

and this'd likely fall into the category of "custom test frameworks" as well.

And in its turn C-test fall into the category not in cargo?

[najamelan]

I would like to propose that this be either reopened, or migrated to libtest. Having to scroll through lots of output in order to figure out whether any tests have failed is a really poor user experience...

[ehuss]

Sure. There is a similar issue, #2832. Cargo would need to use the structured JSON output from libtest (rust-lang/rust#46450), and somehow re-render the output. Unfortunately I don't know of any progress being made to stabilize that or how difficult it would be to use it.

[pickfire]

Is it possible to only show tests that have at least one test running when using a filter?

[FilipAndersson245]

Any update on this issue?

[ClementWalter]

new to rust here, this is a real pain when coming from (more mature ?) languages such as python of javascript. jest/mocha or pytest display a much nicer log when running a project test suite. The cargo one is barely followable and I'm never sure if the test passes without scrolling back to top.

To work on this, where should we start?

[epage]

@ClementWalter at this time, this is blocked on rust-lang/rust#46450. I am starting to experiment with a new json format and will be talking to the libs team about it to see get an idea of what the path for stablization will be.

[chevdor]

This is not near a solution but in the meantime, the following snippet should help reducing the output to a minimum:

function ctest() {
    CL=always
    cargo test --color=$CL $@ 2>/dev/null | grep --color=$CL -v 'running' | grep --color=$CL -v 'test result' | grep --color=$CL -v '^$' 
}

Then instead of:

cargo test [my options]

use:

ctest [my options]

Or even more extreme:

ctest -q [my options]

[ds-cbo]

A slightly more robust version of the helper by @chevdor which doesn't ignore tests with "running" in their name 😛

function ctest() {
  cargo test "$@" |& grep -Ev '^(running [0-9]+ tests?$|test result: ok\.|\s+Running (unittests|tests/)|$)' 
}

[solaeus]

Here's my command to filter the cargo test output down to a useful summary using ripgrep:

cargo test --no-fail-fast | rg --color never '^test|^Running [unittest|test]|^Doc-tests'

I tried to make it unambiguous so that it will work regardless of the test names. It doesn't show why tests fail, it shows a summary of all test results. I use fish but I ran this through sh and it works there as well.

If you use fish, you can add this to to config.fish to get a command that passes its arguments to cargo test:

function cargo_test_summary
    cargo test $argv | rg --color never '^test|^Running [unittest|test]|^Doc-tests'
end

Now you can run cargo_test_summary <test_name> or cargo_test_summary --workspace --no-fail-fast.

[mtalexan]

@ClementWalter at this time, this is blocked on rust-lang/rust#46450. I am starting to experiment with a new json format and will be talking to the libs team about it to see get an idea of what the path for stablization will be.

Was this blocked in some non-obvious way? The specific PR that was referenced was merged in 2018, more than 5 years before this comment saying it was a blocker. Is there some other work that needs to be completed externally for the S-blocked-externally tag?

[weihanglo]

JSON output format is not stabilized yet. See rust-lang/rfcs#3558 for the stabilization plan, and rust-lang/testing-devex-team#1 for the ongoing works.