MSBuild with target netfx48 temp files present fails with Missing Method.

[Wraith2]

Issue Description

Using a roslyn CodeAnalysis project running on net48 attempting to load a project also targetting net48 will fail to load dependencies caused by a missing method Method not found: 'System.ReadOnlySpan`1<Char> Microsoft.IO.Path.GetFileName(System.ReadOnlySpan`1<Char>)' if the target project has already been compiled and has bin/obj directories present.

Steps to Reproduce

CodeAnalysisApp2.zip
Repro is attached. To use

1. unzip and open the CodeAnalysisApp2.sln file then run it. It should load the console app solution and correctly run through to the console readline without displaying any problems.
2. open the ConsoleApp2.sln and compile or run it
3. reopen the CodeAnalysisApp2.sln and run it again, you should get an error:

Using MSBuild at 'C:\Program Files\Microsoft Visual Studio\2022\Professional\MSBuild\Current\Bin' to load projects.
Loading solution '..\..\..\..\ConsoleApp2.sln'
Evaluate        0:00.2464048    ConsoleApp2.csproj
Msbuild failed when processing the file 'D:\Programming\csharp10\CodeAnalysisApp2\ConsoleApp2\ConsoleApp2.csproj' with message: Method not found: 'System.ReadOnlySpan`1<Char> Microsoft.IO.Path.GetFileName(System.ReadOnlySpan`1<Char>)'.
Finished loading solution '..\..\..\..\ConsoleApp2.sln'

Versions & Configurations

Latest VS 17.3 update, this worked on 17.2.
MSBuild version 17.3.0+f67e3d35e for .NET Framework 17.3.0.37102

[Forgind]

This is definitely funky! I could reproduce this perfectly. I fully closed VS afterwards then reopened CodeAnalysisApp2.sln, and it failed again, despite having succeeded when I ran it the first time. I also tried just running CodeAnalysisApp2 twice, and it succeeded both times. To me, that means building ConsoleApp2.sln somehow corrupts some local file cache that affects CodeAnalysisApp2.sln, but I'm not sure yet what exactly.

[tmeschter]

I've also seen this. I have VS 17.4.0 Preview 2.0 (specifically, main 32815.74) and the executable is using the Microsoft.Build.Locator package to try to load a project via ProjectCollection.LoadProject. Using the debugger I've verified that the process is picking up Microsoft.Build.dll from C:\Program Files\Microsoft Visual Studio\2022\main\MSBuild\Current\Bin\Microsoft.Build.dll which is what I would expect.

Note that it does not occur when Microsoft.Build.Locator loads MSBuild from a 17.1 Preview 4 build that I happened to have on my system. Note that I have a number of VS installs and it is not clear which one Microsoft.Build.Locator will load from. :-)

[Wraith2]

As I noted in the original report it worked fine on 17.2 and broke on 17.3. I wrote the project one day and had it working, i updated vs when it closed and then the next day it started to fail. It was quite a surprise.

[rainersigwald]

Thanks for the repro project, that plus another nice repo from a Microsoft-internal person really helped me understand what's going on here, which is this:

1. Roslyn 4.2 has (transitive) dependencies on System.Memory 4.0.1.1 (from package version 4.5.4).
2. MSBuild 17.2 has (transitive) dependencies on System.Memory 4.0.1.1 (from package version 4.5.4).
3. MSBuild 17.3 has dependencies on System.Memory 4.0.1.2 (from package version 4.5.5).
4. Since those versions differ, the strong names of the assemblies differ, so the .NET Framework is happy to load them side by side (this would be different on .NET 6).
5. At runtime, Locator correctly hooks up its assembly resolve events.
6. Then Roslyn stuff or the app itself call some method that causes System.Memory 4.0.1.1 to be loaded.
7. Then MSBuild is called and resolved through Locator's event handlers.
8. Then MSBuild tries to call System.Memory 4.0.1.2, which is located and loaded by Locator.
9. Everything is great! The assemblies are side-by-side but don't try to interchange types so everything is happy.
10. Then MSBuild tries to call this method:

msbuild/src/Shared/FileMatcher.cs

Lines 1653 to 1663 in 92e0776

	internal static bool IsFileNameMatch(string path, string pattern)
	{
	// Use a span-based Path.GetFileName if it is available.
	#if FEATURE_MSIOREDIST
	return IsMatch(Microsoft.IO.Path.GetFileName(path.AsSpan()), pattern);
	#elif NETSTANDARD2_0
	return IsMatch(Path.GetFileName(path), pattern);
	#else
	return IsMatch(Path.GetFileName(path.AsSpan()), pattern);
	#endif
	}

which compiles down to this IL for our net472 target:

    IL_0000: ldarg.0
    IL_0001: call valuetype [System.Memory]System.ReadOnlySpan`1<char> [System.Memory]System.MemoryExtensions::AsSpan(string)
    IL_0006: call valuetype [System.Memory]System.ReadOnlySpan`1<char> [Microsoft.IO.Redist]Microsoft.IO.Path::GetFileName(valuetype [System.Memory]System.ReadOnlySpan`1<char>)
    IL_000b: ldarg.1
    IL_000c: call bool Microsoft.Build.Shared.FileMatcher::IsMatch(valuetype [System.Memory]System.ReadOnlySpan`1<char>, string)
    IL_0011: ret

11. This throws System.MissingMethodException: 'Method not found: 'System.ReadOnlySpan`1<Char> Microsoft.IO.Path.GetFileName(System.ReadOnlySpan`1<Char>)'.'
12. But why? We have a System.Memory of both versions loaded!
13. Microsoft.IO.Redist.dll depends on System.Memory 4.0.1.1 (it is unchanged but System.Memory has a new version)
14. So its return type is really [System.Memory 4.0.1.1] System.ReadOnlySpan`1<char>, but the type we need in Microsoft.Build is a [System.Memory 4.0.1.2] System.ReadOnlySpan`1<char> (because that was the winning version in our build).
15. That's a mismatch, so the MissingMethodException is thrown: no method found matching that (version of) that type in its signature.
16. This doesn't fail in normal MSBuild.exe operation, because we have binding redirects in place that ensure that all System.Memory types are 4.0.1.2:

msbuild/src/MSBuild/app.amd64.config

Lines 85 to 86 in 92e0776

	<assemblyIdentity name="System.Memory" publicKeyToken="cc7b13ffcd2ddd51" culture="neutral" />
	<bindingRedirect oldVersion="0.0.0.0-4.0.1.2" newVersion="4.0.1.2" />

17. In the client application, there is no such binding redirect, because it's compiled against the .2 versions of everything that don't need one.

Tactically, there are two fixes that can be applied:

1. Update the application's reference to the System.Memory to version 4.5.5 and binding redirect to it
   a. This may require adding the reference explicitly; it was likely transitive before
   b. Updating MSBuild references to 17.3 will do this automatically since that package depends on 4.5.5.
   c. You probably don't need to do binding redirects manually because they should be automatic
2. Remove System.Memory.dll from the application directory
   a. This causes Locator to find the System.Memory that lives in MSBuild, which will be the correct version for MSBuild.
   b. If that version is the only version that can be loaded, the .NET Framework (essentially) binding-redirects all references to that version, so everything sees 4.0.1.2 and is consistent
   c. This can be nontrivial since it's coming to the app's output folder as a transitive reference.
   d. This also requires that MSBuildLocator be called before using any System.Memory types, which is stricter than was required before.

[Epic-Santiago]

I have tested and confirmed that doing a reverse bindingRedirect that forces the newer (4.5.5) version of System.Memory to an older version (4.5.4) resolves this issue. Obviously this is a workaround and not a proper solution but it may help someone else.

<configuration>
  <runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <dependentAssembly>
        <assemblyIdentity name="System.Memory" publicKeyToken="cc7b13ffcd2ddd51" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-4.0.1.2" newVersion="4.0.1.1" />
      </dependentAssembly>
    </assemblyBinding>
  </runtime>
</configuration>

[Wraith2]

I've also confirmed that adding a direct reference to the 4.5.5 version of System.Memory resolves the issue.
Do you want me to leave this open issue to track a fix rather than the workarounds given?

[rainersigwald]

Let's leave it open for now to see if it's possible to fix in the MSBuild layer--but right now I don't think it is.

[Epic-Santiago]

I've also confirmed that adding a direct reference to the 4.5.5 version of System.Memory resolves the issue.

Explicitly adding a System.Memory PackageReference was needed because the transitive dependency was broken. This was fixed in Microsoft.Build 17.3 and it's a cleaner fix to upgrade that package than to add the explicit System.Memory reference.

[Wraith2]

How do I get that version of Microsoft.Build? it's not a dependency of the project and I'm on the latest stable visual studio release which still has the problem unless I add the direct dependency.

[rainersigwald]

If you use MSBuildWorkspace, it's a transitive reference of that so you can either get a new version with a direct reference to Microsoft.Build 17.3.1 or add a direct reference to System.Memory 4.5.5. If you don't have a direct reference to MSBuild I'd just do System.Memory, personally.

[AArnott]

This just broke the tests running locally and in Azure Pipelines for Nerdbank.GitVersioning. :(
https://dev.azure.com/andrewarnott/OSS/_build/results?buildId=6954&view=logs&j=0bc77094-9fcd-5c38-f6e4-27d2ae131589&t=206655b7-7ebc-5b58-0ce4-544602a53fe7

Adding the reference to System.Memory to my test project resolved the problem for me. Upgrading my MSBuild reference wasn't going to work out because my test project targets net461.

[zivkan]

FYI, this is affecting nuget.exe as well.

We had to skip a bunch of our tests that no longer work: NuGet/NuGet.Client#4768
And we now have a customer telling us it's impacting them: NuGet/Home#12165

[Forgind]

Were either of rainersigwald's mitigations helpful:
#7873 (comment)
?