dotnet test should fail when it can't find any tests

[wli3]

---

Issue moved from dotnet/sdk#13942

• Please respond to @omajid.

---

From @omajid on Thursday, October 1, 2020 7:26:04 PM

At various times, dotnet test seems to run into an issue where it can't identify any tests. When that happens, dotnet test exits with a 0 exit code. Everything interprets 0 as success. That tells the user (and any automated systems) that everything is working, even though no tests were found which should actually be an error.

This happened in .NET 5 Preview 8:

Running xUnit tests fails to discover tests in with message No test is available in Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.. To solve this update your xunit.runner.visualstudio package to version 2.4.3. This version is used in dotnet new templates, but existing projects targetting, or updating to net5.0 need to be updated manually.

I tried migrating to .NET 5 Preview 8and all my tests "passed", because the tests couldn't be found at all.

Would it be possible to make dotnet test fail with an non-zero exit code when tests can't be found? This makes it easier for developers to catch issues in their CI instead of incorrectly assuming that all tests are passing. When I see that tests fail, I can take steps to find the error and apply the workarounds.

[wli3]

---

Issue moved from dotnet/sdk#13942

• Please respond to @omajid.

---

From @omajid on Thursday, October 1, 2020 7:26:13 PM

cc @tmds @aslicerh

[wli3]

---

Issue moved from dotnet/sdk#13942

• Please respond to @omajid.

---

From @omajid on Thursday, October 1, 2020 7:32:41 PM

We ran into this issue during a previous release too: #2079. This also had the bug that dotnet test was not actually working but returning with a 0 exit code fooling CI into thinking that everything was working correctly.

[wli3]

---

Issue moved from dotnet/sdk#13942

• Please respond to @tmds.

---

From @tmds on Friday, October 2, 2020 7:22:20 AM

A zero exit code MUST mean success for dotnet test.

I feel like clicking the ':+1:'-button a hundred times.

cc @KathleenDollard

[nohwnd]

#2247
#2262

Those are related. I don't think we can make it the default, but we could add an option for it.

[omajid]

Those are related. I don't think we can make it the default, but we could add an option for it.

That doesn't solve the original problem, does it?