Consider no matching named test a failure

[sbeckeriv]

Hello,
I was running a test to watch it fail but it was not. It turns out I typed the test wrong. It seems like that if a test name is provided but not found it should be a failure or at least a warning.
cargo test test_something_that_is_typo

Thoughts?
Becker

Closed: rust-lang/cargo#2410

[bluss]

That's tricky to fix in just libtest too, since cargo test can possibly run multiple test runners, and you would want tests to fail if none of the runners have a match, it's annoying if it would fail if any runner was without match.

[sbeckeriv]

I understand the multi-test framework case would be very annoying. I finally got to testing this in rspec to see what they do and they pass the test but print out "All examples were filtered out". I need to double check the output again but I think the warning would be a nice solution.

[bluss]

It's enough that you have more than one file in tests/ for it to be annoying.

[djrenren]

Just ran into this case myself. It makes scripting the test commands really a pain. Couldn't cargo only issue the success code if at least one of its targets found a match and they passed? Either that or allow for a command line flag like --require-match or something like that.

[Mark-Simulacrum]

I think the best thing to do here is to add N filtered out to the test result line.

Here are some instructions on what to do to fix this, but it's probably best to wait on others' approval before working on this, since it's not entirely decided this is the right approach to take. However, if you'd like to work on this, this should be what needs to be changed in libtest. Once you've done these, please open a new PR with your changes! Let us know if you need any assistance either here or in #rust-internals on irc.mozilla.org.

• A difference between the length of the tests vector before and after this line.
• Store that in the test data struct by adding filtered_tests: usize here.
• And then use that here.

@rust-lang/libs: Could you decide if the approach above seems like the best idea?

[Mark-Simulacrum]

After discussion on IRC, since there's probably some solution here, feel free to take this! Directions above, let us know if you need anything to help make this happen.

[mersinvald]

@Mark-Simulacrum I'd like to take that, please assign me

[Mark-Simulacrum]

Sounds good! I don't think I can assign you, but it's yours. Let me know if you need any help, and please cc me on the pull request when you open it. Let me know if you need any help!

[mersinvald]

@Mark-Simulacrum how can I install compiler and cargo with my changes to make sure cargo test works as expected?

[Mark-Simulacrum]

@mersinvald You shouldn't need to update cargo, so you can specify something like RUSTC=build/x86_64-unknown-linux-gnu/stage1/bin/rustc cargo test. You may need to change the triple (x86_64..), but it should be in the build directory. Let me know if something I've said doesn't work.

[mersinvald]

@Mark-Simulacrum do I need stage2 compiler or stageI will be enough?

[Mark-Simulacrum]

@mersinvald Stage 1 should be enough, but if it doesn't work, go with stage 2.. I'm not 100% sure when it's pulled in. ./x.py build --stage 1 src/libtest should be sufficient, if stage 1, otherwise specify --stage 2.

[mersinvald]

@Mark-Simulacrum thanks for your help!
I couldn't run cargo test because of errors with compiler version incompatibility (is it OK that cargo uses my rustup rustc to build tests?):

error: aborting due to previous error

thread 'rustc' panicked at 'Box<Any>', /checkout/src/librustc_errors/lib.rs:457
thread 'rustc' panicked at 'couldn't compile the test', /checkout/src/librustdoc/test.rs:275

---- src/lib.rs -  (line 6) stdout ----
	error[E0514]: found crate `reed_solomon` compiled by an incompatible version of rustc
 --> <anon>:1:1
  |
1 | extern crate reed_solomon;
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
  = help: please recompile that crate using this compiler (rustc 1.19.0-nightly (f1140a331 2017-05-08))
  = note: crate `reed_solomon` path #1: /home/mike/dev/rust/reed-solomon-rs/target/debug/deps/libreed_solomon-52af978598229f66.rlib compiled by "rustc 1.19.0-dev (bb8d51c2e 2017-05-10)"

But I managed to build tests binary with cargo rustc -- --test and it runs:

$ ./test-binary gf::tests::sub
running 1 test
test gf::tests::sub ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 24 filtered out