-
Notifications
You must be signed in to change notification settings - Fork 253
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Microsoft.TestPlatform (17.4.0-preview-20220726-02) not working #3255
Comments
Thanks for the report does this happen on any test project of just one specific test project? |
Several different projects in our solution. Not all, but certainly not one specific one. |
This just occurred on our Azure Pipelines. Only thing to change is the version Microsoft (R) Test Execution Command Line Tool Version 17.4.0 (x64) |
Does it fail again on 17.4.0? |
We have the same problem. Be good to know if there is any solution. |
@brandscill which exact version are you having issues with please? |
@nohwnd 17.4.1 |
Started facing something similar.
|
@GoAvalon Can you double check your Microsoft.XYZ.Tests.TestEnvironment.AssemblyInitialize method, does it meet what is described in the error? What version of testing framework are you using? |
@nohwnd Only with vstest 17.5 in our build pipelines we see this error that I mentioned above. |
@GoAvalon do you use mstest or xunit or nunit? Which version? :) |
@nohwnd We are using MSTest version 2.1.1 |
I am doing this, and it works (2.1.1. was not working with 17.3.0, but 17.5.0 works). <!-- file mstest68.csproj -->
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netcoreapp3.1</TargetFramework>
<IsPackable>false</IsPackable>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.5.0" />
<PackageReference Include="MSTest.TestAdapter" Version="2.1.1" />
<PackageReference Include="MSTest.TestFramework" Version="2.1.1" />
<PackageReference Include="coverlet.collector" Version="3.1.2" />
</ItemGroup>
</Project>
// file UnitTest1.cs
using Microsoft.VisualStudio.TestTools.UnitTesting;
using System;
namespace mstest68
{
[TestClass]
public class UnitTest1
{
[AssemblyInitialize]
public static void AssemblyInitialize(TestContext context)
{
Console.WriteLine("AssemblyInitialize");
}
[TestMethod]
public void TestMethod1()
{
}
}
} Any suggestions how to repro? |
@nohwnd To fix this I removed the test project reference and now vstest step succeeds. |
Hey there, What's the status here? Is there still some issue going-on? |
I have a repro for this using MSTest 3.4.3 and VSTest 17.10.0 Basically what @GoAvalon said. Steps to reproduce the issue
|
Moved the issue here, it is problem with MSTest, and possibly some conflict with vstest. @Evangelink please try if you can repro. |
Yes I do repro the issue @nohwnd. As discussed offline yesterday, output is good if I use |
Ah that was this issue :D |
I've investigated this with @Evangelink, he noticed that your wildcard pattern is including the TestBase library in both your TestBase/bin and AnotherTestAssembly/bin. Running the dll twice is probably not intended and avoiding this will stop the failure from happening. This is also the reason why I can't repro, I am using full paths and not wildcards (I though you are "anonymizing" your path 😅). And this is why you don't see the same issue in VS, because there you will run with the 2 exact paths that you get from build, not with a wildcard. We've investigated this to find the root cause. The failure itself is because when inspecting the dll MSTest/v1 will load due to the rules in the custom assembly resolver in VSTest, this will try to run the tests as MSTest/v1 tests and will fail to assign the property because the types are not compatible. This is why we could not reproduce this in standalone vstest.console, and only in the on that is under VS installation. You don't have to use this workaound, but new vstest (17.11) comes with a runsetting that is |
This is just the fastest repro I came up with. Our actual situation is that we have a Tests folder, where we build every test project in a subfolder and then just glob all test DLLs via The solution for us was the same as GoAvalon's. We extracted the shared code to a library without any tests, thus avoiding mutliple copies of an assembly with tests ending up in our build output.
Just for reference, I found the change you mentioned here: microsoft/vstest#5018 |
That is a good solution, I think it is good practice to not mix test project, and test infrastructure projects. You can also still reference MSTest in your test infrastructure projects, but it is a good idea to make all test base classes |
Description
Since updating to the latest version of Microsoft.TestPlatform, unittests don't run anymore in Azure DevOps pipelines.
It seems as if there are some assembly reference mismatches, and test attributes are no longer recognized.
https://www.nuget.org/packages/Microsoft.TestPlatform
17.4.0-preview-20220726-02 <== error
17.2.0 (latestStable) <== works
17.4.0-preview-20220707-01 <== works
Steps to reproduce
Build pipeline step:
Expected behavior
Tests running fine.
Actual behavior
example 1:
The test class in question is fine:
example 2:
the method looks fine:
example 3:
the ExpectedException attibute is not honored:
Diagnostic logs
failing output:
working output:
Environment
Azure Devops pipelines
Workarouds:
use earlier versions:
or
The text was updated successfully, but these errors were encountered: