Dependency Update from SymUploader is failing

[epananth]

• This issue is blocking
• This issue is causing unreasonable pain

Got the latest update from SymUploader and that is failing CI

arcade build - https://dnceng.visualstudio.com/internal/_build/results?buildId=947162&view=logs&j=fa59fe4e-195c-56fa-189b-58fd241f10dd&t=afa1ce5e-d2bc-56f3-ad78-7883f25a3e3a

2021-01-12T22:38:53.0342658Z F:\workspace\_work\1\s\.dotnet\sdk\5.0.100\Microsoft.Common.CurrentVersion.targets(2123,5): error MSB3277: Found conflicts between different versions of "System.Collections.Immutable" that could not be resolved.  These reference conflicts are listed in the build log when log verbosity is set to detailed. [F:\workspace\_work\1\s\src\Microsoft.DotNet.Build.Tasks.Feed.Tests\Microsoft.DotNet.Build.Tasks.Feed.Tests.csproj]
2021-01-12T22:38:53.0367753Z ##[error].dotnet\sdk\5.0.100\Microsoft.Common.CurrentVersion.targets(2123,5): error MSB3277: (NETCORE_ENGINEERING_TELEMETRY=Build) Found conflicts between different versions of "System.Collections.Immutable" that could not be resolved.  These reference conflicts are listed in the build log when log verbosity is set to detailed.
2021-01-12T22:38:53.0371550Z ##[debug]Processed: ##vso[task.logissue type=error;sourcepath=F:\workspace\_work\1\s\.dotnet\sdk\5.0.100\Microsoft.Common.CurrentVersion.targets;linenumber=2123;columnnumber=5;code=MSB3277;](NETCORE_ENGINEERING_TELEMETRY=Build) Found conflicts between different versions of "System.Collections.Immutable" that could not be resolved.  These reference conflicts are listed in the build log when log verbosity is set to detailed.
2021-01-12T22:38:53.0374822Z F:\workspace\_work\1\s\.dotnet\sdk\5.0.100\Microsoft.Common.CurrentVersion.targets(2123,5): error MSB3277: Found conflicts between different versions of "System.Collections.Immutable" that could not be resolved.  These reference conflicts are listed in the build log when log verbosity is set to detailed. [F:\workspace\_work\1\s\src\Microsoft.DotNet.Build.Tasks.Feed.Tests\Microsoft.DotNet.Build.Tasks.Feed.Tests.csproj]
2021-01-12T22:38:53.0381078Z ##[error].dotnet\sdk\5.0.100\Microsoft.Common.CurrentVersion.targets(2123,5): error MSB3277: (NETCORE_ENGINEERING_TELEMETRY=Build) Found conflicts between different versions of "System.Collections.Immutable" that could not be resolved.  These reference conflicts are listed in the build log when log verbosity is set to detailed.
2021-01-12T22:38:53.0383789Z ##[debug]Processed: ##vso[task.logissue type=error;sourcepath=F:\workspace\_work\1\s\.dotnet\sdk\5.0.100\Microsoft.Common.CurrentVersion.targets;linenumber=2123;columnnumber=5;code=MSB3277;](NETCORE_ENGINEERING_TELEMETRY=Build) Found conflicts between different versions of "System.Collections.Immutable" that could not be resolved.  These reference conflicts are listed in the build log when log verbosity is set to detailed.
2021-01-12T22:38:53.3874939Z   Microsoft.DotNet.ApiCompat.Tests -> F:\workspace\_work\1\s\artifacts\bin\Microsoft.DotNet.ApiCompat.Tests\Release\net472\Microsoft.DotNet.ApiCompat.Tests.dll
2021-01-12T22:38:53.4418889Z   Microsoft.DotNet.Build.Tasks.Feed.Tests -> F:\workspace\_work\1\s\artifacts\bin\Microsoft.DotNet.Build.Tasks.Feed.Tests\Release\netcoreapp2.1\Microsoft.DotNet.Build.Tasks.Feed.Tests.dll
2021-01-12T22:38:53.8508111Z   Microsoft.DotNet.Deployment.Tasks.Links -> F:\workspace\_work\1\s\artifacts\bin\Microsoft.DotNet.Deployment.Tasks.Links\Release\net472\Microsoft.DotNet.Deployment.Tasks.Links.dll
2021-01-12T22:38:53.8524615Z   Microsoft.DotNet.RemoteExecutor.Tests -> F:\workspace\_work\1\s\artifacts\bin\Microsoft.DotNet.RemoteExecutor.Tests\Release\netcoreapp2.1\Microsoft.DotNet.RemoteExecutor.Tests.dll

cc: @riarenas

[epananth]

@mikem8361 there is some version mismatch error here.. Any idea about this?

[mikem8361]

I'm looking into it. The symbol uploader hasn't been updated in a long while and there have been some changes, but it probably it's symstore repo (which has also changed) dependency that may cause this problem.

[mikem8361]

@epananth, how do you know this is the symbol uploader package? And there are two: Microsoft.SymbolUploader.Build.Task and Microsoft.SymbolUploader (I didn't know arcade actually uses this one). They both depend on System.Collections.Immutable indirectly through the symstore repo indirectly through it's System.Reflection.Metadata reference. Neither repro references it directly. The netstandard2.0 framework (used when running on netcore) references System.Reflection.Metadata version 1.8.1; on net45 (desktop framework) version 1.6.0. That changed two months ago from 1.5.0 and I'm pretty arcade took updates after that. I'm not sure what version of System.Collections.Immutable that 1.8.1 S.R.M references. It is confusing and hard to get the package reference/dependency graph right in all cases.

Issue #6784 looks like it might be caused by a Symbol uploader change for 2GB symbol files that happened mid November where you reverted back to version 1.1.156702. Is this the version this build/reference problem is happening with?

Enabling the verbose logging in the above build def would be helpful.

/cc: @hoyosjs

[hoyosjs]

Arcade stopped using the Task and directly wraps the object now. That pipeline's last commit is going from

Microsoft.Symbol.Uploader@1.1.152002 -> Microsoft.Symbol.Uploader@1.1.206105

There was a conflict between "System.Collections.Immutable, Version=1.2.3.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" and "System.Collections.Immutable, Version=1.2.5.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a".
        "System.Collections.Immutable, Version = 1.2.3.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" was chosen because it was primary and "System.Collections.Immutable, Version=1.2.5.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" was not.
        References which depend on "System.Collections.Immutable, Version = 1.2.3.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" [F:\workspace\_work\1\s\.packages\microsoft.netcore.app\2.1.0\ref\netcoreapp2.1\System.Collections.Immutable.dll].
        F:\workspace\_work\1\s\.packages\microsoft.netcore.app\2.1.0\ref\netcoreapp2.1\System.Collections.Immutable.dll
              Project file item includes which caused reference "F:\workspace\_work\1\s\.packages\microsoft.netcore.app\2.1.0\ref\netcoreapp2.1\System.Collections.Immutable.dll".
                F:\workspace\_work\1\s\.packages\microsoft.netcore.app\2.1.0\ref\netcoreapp2.1\System.Collections.Immutable.dll
            F:\workspace\_work\1\s\artifacts\bin\Microsoft.Arcade.Test.Common\Release\netcoreapp2.1\Microsoft.Arcade.Test.Common.dll
              Project file item includes which caused reference "F:\workspace\_work\1\s\artifacts\bin\Microsoft.Arcade.Test.Common\Release\netcoreapp2.1\Microsoft.Arcade.Test.Common.dll".
                F:\workspace\_work\1\s\artifacts\bin\Microsoft.Arcade.Test.Common\Release\netcoreapp2.1\Microsoft.Arcade.Test.Common.dll
            F:\workspace\_work\1\s\artifacts\bin\Microsoft.DotNet.Build.Tasks.Feed\Release\netcoreapp2.1\Microsoft.DotNet.Build.Tasks.Feed.dll
              Project file item includes which caused reference "F:\workspace\_work\1\s\artifacts\bin\Microsoft.DotNet.Build.Tasks.Feed\Release\netcoreapp2.1\Microsoft.DotNet.Build.Tasks.Feed.dll".
                F:\workspace\_work\1\s\artifacts\bin\Microsoft.DotNet.Build.Tasks.Feed\Release\netcoreapp2.1\Microsoft.DotNet.Build.Tasks.Feed.dll
            F:\workspace\_work\1\s\artifacts\bin\Microsoft.DotNet.Internal.DependencyInjection.Testing\Release\netcoreapp2.1\Microsoft.DotNet.Internal.DependencyInjection.Testing.dll
              Project file item includes which caused reference "F:\workspace\_work\1\s\artifacts\bin\Microsoft.DotNet.Internal.DependencyInjection.Testing\Release\netcoreapp2.1\Microsoft.DotNet.Internal.DependencyInjection.Testing.dll".
                F:\workspace\_work\1\s\artifacts\bin\Microsoft.DotNet.Internal.DependencyInjection.Testing\Release\netcoreapp2.1\Microsoft.DotNet.Internal.DependencyInjection.Testing.dll
        References which depend on "System.Collections.Immutable, Version = 1.2.5.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" [].
    Unified Dependency "System.Collections.Immutable, Version=1.2.5.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a".
        Using this version instead of original version "1.2.3.0" in "F:\workspace\_work\1\s\artifacts\bin\Microsoft.Arcade.Test.Common\Release\netcoreapp2.1\Microsoft.Arcade.Test.Common.dll" because AutoUnify is 'true'.
        Using this version instead of original version "1.2.3.0" in "F:\workspace\_work\1\s\artifacts\bin\Microsoft.DotNet.Internal.DependencyInjection.Testing\Release\netcoreapp2.1\Microsoft.DotNet.Internal.DependencyInjection.Testing.dll" because AutoUnify is 'true'.
        Could not resolve this reference. Could not locate the assembly "System.Collections.Immutable, Version = 1.2.5.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a". Check to make sure the assembly exists on disk. If this reference is required by your code, you may get compilation errors.
        Required by "F:\workspace\_work\1\s\artifacts\bin\Microsoft.Arcade.Test.Common\Release\netcoreapp2.1\Microsoft.Arcade.Test.Common.dll".
        Required by "F:\workspace\_work\1\s\artifacts\bin\Microsoft.DotNet.Build.Tasks.Feed\Release\netcoreapp2.1\Microsoft.DotNet.Build.Tasks.Feed.dll".
        Required by "F:\workspace\_work\1\s\artifacts\bin\Microsoft.DotNet.Internal.DependencyInjection.Testing\Release\netcoreapp2.1\Microsoft.DotNet.Internal.DependencyInjection.Testing.dll".

[epananth]

I think sometime in Nov we stopped using Buildtask and started using the API for symbol upload. We have been using this version 1.1.152002, till we got the latest update. To be sure I ran the arcade CI with the latest version change, that's when I caught the version mismatch error. Earlier it did not throw any error cos I had the dependency name incorrect, while looking at the dependency update I found the mistake and corrected it yesterday #6785.

[hoyosjs]

1.2.5.0 comes from System.Collections.Immutable 1.7.1 (ns2.0)
1.2.3.0 comes from System.Collections.Immutable 1.5.0 (ns2.0)

[hoyosjs]

eng/BuildTask.targets
60:    <PackageReference Update="System.Collections.Immutable" PrivateAssets="All" Publish="false" ExcludeAssets="runtime" />

src/Microsoft.DotNet.Arcade.Sdk/Microsoft.DotNet.Arcade.Sdk.csproj
30:    <PackageReference Include="System.Collections.Immutable" Version="$(SystemCollectionsImmutableVersion)" />
src/Microsoft.DotNet.Build.Tasks.Packaging/tests/Microsoft.DotNet.Build.Tasks.Packaging.Tests.csproj
29:    <TestPackage Include="System.Collections.Immutable" Version="1.5.0" />

eng/Versions.props
51:    <SystemCollectionsImmutableVersion>1.5.0</SystemCollectionsImmutableVersion>

System.Reflection.Metadata 1.6.0 references Immutable 1.5.0 @mikem8361; Still not sure where the 1.2.5.0 reference is coming from. It looks like it might be coming from netcoreapp 3.1, but I am still not sure.

[mikem8361]

netstandard2.0 symbol uploader references symstore which references 1.8.1 version of System.Reflection.Metadata which references 1.7.1 S.C.I.

[hoyosjs]

Somehow I couldn't see that listed as a transitive dependency. @epananth do you know if it's possible to use version 1.7.1?

[epananth]

Need to test and get back to you on this..

[epananth]

I did a test against arcade, created a PR -> #6792

Want to test against arcade-validation to see if everything works..

[epananth]

Update on this, the change worked but this broke a bunch of tests and made some tests flaky, not sure how to fix them. So I asked @ViktorHofer to help me with this..

[ViktorHofer]

I re-enabled the subscription and the PR was created to update symuploader: #7489. Let's see what breaks.

cc @hoyosjs @adiaaida

[ViktorHofer]

WOHOO, PR is through. Closing.