dotnet test does not respect verbosity on CircleCI

[guitarrapc]

Description

IMPORTANT: if the defect is reproduced only in a workflow from within the Visual Studio IDE then do not report the issue here - instead, please report it using Visual Studio's "Send Feedback" option that can be accessed from the Help menu OR using this link https://developercommunity.visualstudio.com.

For a defect reproducable from the vstest command line, describe the issue you've observed.

When running dotnet test on CircleCI, some how it's default verbosity was recognized as quiet, normally default verbosity is minimal.

Even if I want to override verbosity, it ignores arguments override --verbosity minimal or "--logger:Console;verbosity=normal".

So, there are 2 problems running dotnet test on CircleCI.

1. dotnet test verbosity level is not respecting default minimal, but it use quiet.
2. dotnet test could not override verbosity level with --verbosity or "--logger:Console;verbosity=minimal".

Steps to reproduce

What steps can reproduce the defect?
Please share the setup, commandline for vstest.console, sample project, target
framework etc.

Here's minimal project to reproduce issue.

https://github.com/guitarrapc/dotnet-test-lab

• create C# xunit project with .NET Core 3.1.
• upgrade nuget package to the latest.
  • Microsoft.NET.Test.Sdk: 16.2.0 -> 16.4.0
• add .circleci/config.yml and configure dotnet build and dotnet test -c Debug "--logger:Console;verbosity=minimal" runs on dotnetsdk:3.1.
• add 1 expected to failed test, something like Assert.False(true);.
• run build on CircleCI
• you will find no failed test detail reported on log.

A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.1 (64-bit .NET Core 3.1.0)
Test run in progress.
Exited with code exit status 1

Expected behavior

Share the expected output

• dotnet test -c Debug "--logger:Console;verbosity=minimal" should override logger verbosity from quiet to minimal, and output failed test detail on circleci.

A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:00.64]     XUnitTestProject1.UnitTest1.BoolFailTest [FAIL]
Test run in progress.  X XUnitTestProject1.UnitTest1.BoolFailTest [2ms]
  Error Message:
   Assert.False() Failure
Expected: False
Actual:   True
  Stack Trace:
     at XUnitTestProject1.UnitTest1.BoolFailTest() in /root/project/tests/XUnitTestProject1/UnitTest1.cs:line 33
Test run in progress.
Test Run Failed.
Total tests: 4
     Passed: 3
     Failed: 1
 Total time: 1.4908 Seconds
VSTest: Exit code: 1

Exited with code exit status 1

Actual behavior

What is the behavior observed?

• dotnet test -c Debug "--logger:Console;verbosity=minimal" could not override logger verbosity from quiet to minimal, and output not contains failed test detail on circleci.

see circleci build log.

https://circleci.com/gh/guitarrapc/dotnet-test-lab/6

A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:00.49]     XUnitTestProject1.UnitTest1.BoolFailTest [FAIL]
Test run in progress.VSTest: Exit code: 1

Exited with code exit status 1

Diagnostic logs

Please share test platform diagnostics logs. Instructions to collect logs are here.
The logs may contain test assembly paths, kindly review and mask those before sharing.

you can retrieve from circleci job.

https://circleci.com/gh/guitarrapc/dotnet-test-lab/8#artifacts/containers/0

• failed diagnostics log:
  • log.txt: https://8-233778155-gh.circle-artifacts.com/0/root/project/bin/default/log.txt
  • log.host.20-01-14_07-57-36_58062_4.txt: https://8-233778155-gh.circle-artifacts.com/0/root/project/bin/default/log.host.20-01-14_07-57-36_58062_4.txt

It seems like verbosity is successfully override with what I expected, but result is not according to verbosity..

Environment

Please share additional details about the test environment.
Operating system, Build version of vstest.console

• CircleCI
• .NET Core 3.1 SDK (docker)
• Microsoft.NET.Test.Sdk: 16.4.0
• xunit.runner.visualstudio: 2.4.1
• xUnit: 2.4.1

[guitarrapc]

Workaround

I can fix this issue by adding < /dev/null to the dotnet test.

before

- run: dotnet test -c Debug "--logger:Console;verbosity=minimal"

after

- run: dotnet test -c Debug "--logger:Console;verbosity=minimal" < /dev/null

also you can omit "--logger:Console;verbosity=minimal" because default verbosity recognized to minimal as expected.

- run: dotnet test -c Debug < /dev/null

See config.yml for the detail.

https://github.com/guitarrapc/dotnet-test-lab/blob/2f7638d35db92d8aa4563d8bf0a8572f0eb9d421/.circleci/config.yml

[guitarrapc]

So the issue is relates to the #2080

[nohwnd]

There was a recent bug fix for case sensitivity of verbosity, maybe this was also affected by it? #2300 could you try with the latest dotnet test? Please also share the few lines of output after you run the command, there is a version of the test platform cli tools that are shipping with the dotnet test. New dotnet test that uses 16.5.0 that has fix for the issue should be available in the upcoming days. Otherwise if you are interested the linked PR is where the code is dealing with verbosity parsing, so the error would be somewhere there. A PR to fix this would be very welcome 🙂

[dominikjeske]

I have same issue on TeamCity. Locally everything works but on build server I have verbosity with debug info in logs. Build tools installed on server have version 16.5.

dotnet test $solution.FullName --no-build --nologo --no-restore -l "console;verbosity=quiet"

[nohwnd]

Seems to be fixed.

[guitarrapc]

thanks a lot!