vstest.console.exe grabs exclusive read access to its test container

[pvvl]

Description

vstest.console.exe uses .NET's System.IO.File.Open() with a default ShareMode argument. As a consequence, attempts by other processes to read the DLL simultaneously will fail.

We use several vstest.console instances on the same DLL for maximal execution speed. This fails occasionally because of this race condition.
vstest.console has its own parallel execution mechanism, but the latter is still slower, offers less isolation, and does not allow distributed execution.

The problem is actually due to an ill-chosen default in the .NET framework. I filed a detailed description about this problem (in combination with vstest.console) entitled: ".NET's System.IO.File.Open() variants have default FileShare.None"

Steps to reproduce

• create a DLL containing a unit test that sleeps for a minute or so.
• start vstest.console to execute that test
• while one test is running, start another vstest.console with the same arguments.

Expected behavior

There is no reason why running two tests in parallel would require exclusive access to the shared test container. In a concurrent environment, this behavior is a recipe for disaster.

Actual behavior

Occasional race conditions occur when executing tests in parallel from the same container.

Diagnostic logs

Whenever the race condition causes a collision, vstest.console produces an error message like this:
Could not load file or assembly 'file:///.dll' or one of its dependencies. The process cannot access the file because it is being used by another process. (Exception from HRESULT: 0x80070020).
I was able to get the call stack where the exclusive access is being requested: VSTEST.CONSOLE is an attemptingto open a test container with the (implicit!) share mode = none.
In the end I caught the problem using a conditional breakpoint on CreateFileW, checking for dwShareMode == 0x0.

I hit this conditional breakpoint with the following call stack:

KernelBase.dll!_CreateFileW@28�() Unknown Symbols loaded.
kernel32.dll!CreateFileWImplementation@28�() Unknown Symbols loaded.
mscorlib.ni.dll!79a7d2f8() Unknown No symbols loaded.
[Frames below may be incorrect and/or missing, native debugger attempting to walk managed call stack] Annotated Frame
mscorlib.ni.dll!79a7d2f8() Unknown No symbols loaded.
[Managed to Native Transition] Annotated Frame
mscorlib.dll!Microsoft.Win32.Win32Native.SafeCreateFile(string lpFileName, int dwDesiredAccess, System.IO.FileShare dwShareMode, Microsoft.Win32.Win32Native.SECURITY_ATTRIBUTES securityAttrs, System.IO.FileMode dwCreationDisposition, int dwFlagsAndAttributes, System.IntPtr hTemplateFile) Line 977 C# Symbols loaded.
mscorlib.dll!System.IO.FileStream.Init(string path, System.IO.FileMode mode, System.IO.FileAccess access = Read, int rights, bool useRights = false, System.IO.FileShare share, int bufferSize = 4096, System.IO.FileOptions options, Microsoft.Win32.Win32Native.SECURITY_ATTRIBUTES secAttrs, string msgPath = "Lms_UnitTests_GPKManagedTestStubs.dll", bool bFromProxy = false, bool useLongPath, bool checkHost) Line 843 C# Symbols loaded.
mscorlib.dll!System.IO.FileStream.FileStream(string path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share) Line 527 C# Symbols loaded.
mscorlib.dll!System.IO.File.Open(string path, System.IO.FileMode mode, System.IO.FileAccess access) Line 474 C# Symbols loaded.
vstest.console.exe!Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.Utilities.AssemblyMetadataProvider.GetFrameWork(string filePath = "n:/TestLab/GPK/output/bin/Win32/Release/Lms_UnitTests_GPKManagedTestStubs.dll") Line 30 C# Symbols loaded.
vstest.console.exe!Microsoft.VisualStudio.TestPlatform.CommandLineUtilities.InferHelper.DetermineFrameworkName(System.Collections.Generic.IEnumerable sources = Count = 1, System.Collections.Generic.IDictionary<string, Microsoft.VisualStudio.TestPlatform.ObjectModel.Framework> sourceFrameworkVersions = Count = 0, out bool conflictInFxIdentifier = false) Line 134 C# Symbols loaded.
vstest.console.exe!Microsoft.VisualStudio.TestPlatform.CommandLineUtilities.InferHelper.AutoDetectFramework(System.Collections.Generic.List sources = Count = 1, System.Collections.Generic.IDictionary<string, Microsoft.VisualStudio.TestPlatform.ObjectModel.Framework> sourceFrameworkVersions = Count = 0) Line 101 C# Symbols loaded.
vstest.console.exe!Microsoft.VisualStudio.TestPlatform.CommandLine.TestPlatformHelpers.TestRequestManager.UpdateRunSettingsIfRequired(string runsettingsXml = "\r\n\r\n \r\n 1\r\n n:\TestLab\GPK\TestResults\r\n x86\r\n Framework45\r\n \r\n \r\n \r\n \r\n \r\n \r\n True\r\n false\r\n True\r\n False\r\n \r\n <Directory Path="%SolutionDir%\output\bin\%Platform_Actual%\%Configuration%" includeSubDirectories="false" />\r\n \r\n \r\n \r\n 1\r\n \r\n n:\TestLab\GPK\r\n", System.Collections.Generic.List sources = Count = 1, out string updatedRunSettingsXml = "\r\n\r\n \r\n 1\r\n n:\TestLab\GPK\TestResults\r\n x86\r\n Framework45\r\n \r\n \r\n \r\n \r\n \r\n \r\n True\r\n false\r\n True\r\n False\r\n \r\n <Directory Path="%SolutionDir%\output\bin\%Platform_Actual%\%Configuration%" includeSubDirectories="false" />\r\n \r\n \r\n \r\n 1\r\n \r\n n:\TestLab\GPK\r\n") Line 367 C# Symbols loaded.
vstest.console.exe!Microsoft.VisualStudio.TestPlatform.CommandLine.TestPlatformHelpers.TestRequestManager.DiscoverTests(Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.DiscoveryRequestPayload discoveryPayload = {Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.DiscoveryRequestPayload}, Microsoft.VisualStudio.TestPlatform.Common.Interfaces.ITestDiscoveryEventsRegistrar discoveryEventsRegistrar = {Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.RunSpecificTestsArgumentExecutor.DiscoveryEventsRegistrar}, Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.ProtocolConfig protocolConfig = {Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.ProtocolConfig}) Line 147 C# Symbols loaded.
vstest.console.exe!Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.RunSpecificTestsArgumentExecutor.DiscoverTestsAndSelectSpecified(System.Collections.Generic.IEnumerable sources = Count = 1) Line 238 C# Symbols loaded.
vstest.console.exe!Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.RunSpecificTestsArgumentExecutor.Execute() Line 214 C# Symbols loaded.
vstest.console.exe!Microsoft.VisualStudio.TestPlatform.CommandLine.Executor.ExecuteArgumentProcessor(Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.IArgumentProcessor processor = {Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.RunSpecificTestsArgumentProcessor}, ref int exitCode = 0) Line 328 C# Symbols loaded.
vstest.console.exe!Microsoft.VisualStudio.TestPlatform.CommandLine.Executor.Execute(string[] args = {string[3]}) Line 131 C# Symbols loaded.
vstest.console.exe!Microsoft.VisualStudio.TestPlatform.CommandLine.Program.Main(string[] args = {string[3]}) Line 39 C# Symbols loaded.
[Native to Managed Transition] Annotated Frame
mscoreei.dll!CorExeMain@0�() Unknown Symbols loaded.
mscoree.dll!ShellShimCorExeMain@0�() Unknown Symbols loaded.
mscoree.dll!CorExeMain_Exported@0�() Unknown Symbols loaded.
ntdll.dll!RtlUserThreadStart@8�() Unknown Symbols loaded.
ntdll.dll!_RtlUserThreadStart@8�() Unknown Symbols loaded.

Environment

File version of vstest.console.exe: 15.0.27617.1. This problem is reproducible in Version 15.8.0-dev of VSTEST.CONSOLE.

[pvvl]

IMHO the problem can be solved by modifying line 30 of VSTest\vstest-master\src\vstest.console\CommandLine\AssemblyMetadataProvider.cs:
changing
using (var assemblyStream = File.Open(filePath, FileMode.Open, FileAccess.Read))
into
using (var assemblyStream = File.Open(filePath, FileMode.Open, FileAccess.Read, FileShare.Read))

Our organization needs these changes to end up on an official Microsoft installation, so a locally patched version of vstest.console does not cut it.

[smadala]

@pvvl Did you tried that the fix works for your scenario? We are happy to accept your contribution 😄

[pvvl]

Hi, Yes, I did. It is somewhat harder to reproduce than I originally thought. The race condition occurs when the following events intertwine: · Instance 1 of vstest.console is opening the test container with exclusive access to determine the framework version (function GetFrameWork ). This typically takes only a short time, and happens within an exception try/catch block. · Instance 2 of vstest.console loads the test container in memory to execute the tests. This can be lengthy. If you start these two processes by hand, then typically the following happens: · Process 1 determines the framework version, and has exclusive access, but only for a short while. Then it proceeds to load the DLL and starts executing tests. · Process 2 tries to determine the framework version and fails, but the exception handler takes care and controls the damage. Then it proceeds to load the DLL and all is fine. However, I can Divinely Interfere ☺ using the debugger, to force the timing into a race condition: · Process 1 runs until the test container is opened for exclusive access in GetFrameWork, then hits a breakpoint. · Process 2 now fails once in trying to determine the framework version in GetFramework, but survives thanks to the exception handler. But afterwards it crashes because loading the DLL in memory will not succeed. Using the latter technique, I managed to reproduce the problem using the unchanged code. With the fix, the problem no longer occurred. Kind regards, Piet Van Vlierberghe From: Satya Madala [mailto:notifications@github.com] Sent: Friday, June 08, 2018 4:18 PM To: Microsoft/vstest <vstest@noreply.github.com> Cc: Van Vlierberghe, Pieter (DF PL STS TEST PRD 1) <pieter.vanvlierberghe@siemens.com>; Mention <mention@noreply.github.com> Subject: Re: [Microsoft/vstest] vstest.console.exe grabs exclusive read access to its test container (#1638) @pvvl<https://github.com/pvvl> Did you tried that the fix works for your scenario? We are happy to accept your contribution 😄 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#1638 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AbCWKzesqOH93OKK4XhfeF5J0fyF7j-eks5t6oeEgaJpZM4Uf1-q>.

[smadala]

@pvvl It makes sense to me. Feel free to raise a PR. We will ship in next release(15.8).

[pvlakshm]

@pvvl, yo mention "We use several vstest.console instances on the same DLL for maximal execution speed."

Have you tried vstest's parallel test execution?
If you are using MSTest V2 based tests, you then try in-assembly parallel execution?

These can significantly reduce the overall time taken to complete a test run.

[pvvl]

Hi, Thanks for the suggestions. Indeed, we parallelize by ourselves because of a number of reasons: · The majority of the tests are unmanaged code, so in-assembly parallel execution is not an option. · We want tests to minimally interfere. Parallel test execution does not guarantee separate processes for each test, as the test processes act as a pool, and execute whatever is thrown at them. So if e.g. one test has a huge memory leak, another test may suffer from it, but we will never be able to tell and the problem will not be reproducible. · By controlling test execution by ourselves, we can easily scale up by distributing the work in a cluster. From: pvlakshm [mailto:notifications@github.com] Sent: Sunday, June 10, 2018 7:46 AM To: Microsoft/vstest <vstest@noreply.github.com> Cc: Van Vlierberghe, Pieter (DF PL STS TEST PRD 1) <pieter.vanvlierberghe@siemens.com>; Mention <mention@noreply.github.com> Subject: Re: [Microsoft/vstest] vstest.console.exe grabs exclusive read access to its test container (#1638) @pvvl<https://github.com/pvvl>, yo mention "We use several vstest.console instances on the same DLL for maximal execution speed." Have you tried vstest's parallel test execution<https://blogs.msdn.microsoft.com/devops/2016/10/10/parallel-test-execution/>? If you are using MSTest V2 based tests, you then try in-assembly parallel execution<https://github.com/Microsoft/testfx-docs/blob/master/RFCs/004-In-Assembly-Parallel-Execution.md>? These can significantly reduce the overall time taken to complete a test run. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#1638 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AbCWKwWgozBTjXboiD6BWGLFXc0rCnJbks5t7LKYgaJpZM4Uf1-q>.

[smadala]

@pvvl Create a #1660 to address this. Please review it. If possible test on your scenario.

@qingjuntian you might well interested on this issue.

[smadala]

Fix for this will be available in VS 15.8