BinaryFormatter tests should be skipped only on AOT, WASM and Mobile #106858

adamsitnik · 2024-08-23T06:43:50Z

A follow up to the discussion we have started at #106737

…e it's not supported by design

dotnet-policy-service · 2024-08-23T06:46:57Z

Tagging subscribers to this area: @dotnet/area-system-runtime
See info in area-owners.md if you want to be subscribed.

…s in the root Directory.Build.props file)

…were being skipped for a while)

adamsitnik · 2024-08-26T10:50:05Z

The initial change I’ve sent with this PR has allowed me to discover that there were 19 projects where BF-related tests were skipped since #103255 was merged.

@ericstj I’ve re-enabled those tests by adding a project reference similar to what you did in #106737. We don’t want to run these tests on Mobile, WASM and NativeAOT (not supported by design). How adding such project reference is going to affect the build time for those configurations? Should we somehow make it conditional? If so, should we introduce an MSBuild property (or any other setting) and handle all that logic is some .targets file so we would not need to copy it everywhere and update all occurrences when changes are needed?

@bartonjs There are 5 System.Collections.Tests tests that fail due to a product change. I think you have mentioned it at some point, but I just want to make sure that we are aware of that change, it’s documented and expected.

    Starting:    System.Collections.Tests (parallel test collections = on [24 threads], stop on fail = off)
      System.Collections.Tests.HashSet_Generic_Tests_string.ComparerSerialization [FAIL]
        Assert.IsType() Failure: Value is not the exact type
        Expected: typeof(System.Collections.Generic.StringEqualityComparer)
        Actual:   typeof(System.Collections.Generic.GenericEqualityComparer<string>)
        Stack Trace:
          D:\projects\runtime\src\libraries\System.Collections\tests\Generic\HashSet\HashSet.Generic.Tests.cs(891,0): at System.Collections.Tests.HashSet_Generic_Tests`1.<ComparerSerialization>g__TestComparerSerialization|64_0[TCompare
  d](IEqualityComparer`1 equalityComparer, String internalTypeName)
          D:\projects\runtime\src\libraries\System.Collections\tests\Generic\HashSet\HashSet.Generic.Tests.cs(859,0): at System.Collections.Tests.HashSet_Generic_Tests`1.ComparerSerialization()
             at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
          D:\projects\runtime\src\libraries\System.Private.CoreLib\src\System\Reflection\MethodBaseInvoker.cs(57,0): at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
      System.Collections.Tests.Dictionary_Tests.ComparerSerialization [FAIL]
        Assert.IsType() Failure: Value is not the exact type
        Expected: typeof(System.Collections.Generic.StringEqualityComparer)
        Actual:   typeof(System.Collections.Generic.GenericEqualityComparer<string>)
        Stack Trace:
          D:\projects\runtime\src\libraries\System.Collections\tests\Generic\Dictionary\Dictionary.Tests.cs(457,0): at System.Collections.Tests.Dictionary_Tests.TestComparerSerialization[T](IEqualityComparer`1 equalityComparer, String
  internalTypeName)
          D:\projects\runtime\src\libraries\System.Collections\tests\Generic\Dictionary\Dictionary.Tests.cs(426,0): at System.Collections.Tests.Dictionary_Tests.ComparerSerialization()
             at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
          D:\projects\runtime\src\libraries\System.Private.CoreLib\src\System\Reflection\MethodBaseInvoker.cs(57,0): at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
      System.Collections.Tests.HashSet_Generic_Tests_int.ComparerSerialization [FAIL]
        Assert.IsType() Failure: Value is not the exact type
        Expected: typeof(System.Collections.Generic.StringEqualityComparer)
        Actual:   typeof(System.Collections.Generic.GenericEqualityComparer<string>)
        Stack Trace:
          D:\projects\runtime\src\libraries\System.Collections\tests\Generic\HashSet\HashSet.Generic.Tests.cs(891,0): at System.Collections.Tests.HashSet_Generic_Tests`1.<ComparerSerialization>g__TestComparerSerialization|64_0[TCompare
  d](IEqualityComparer`1 equalityComparer, String internalTypeName)
          D:\projects\runtime\src\libraries\System.Collections\tests\Generic\HashSet\HashSet.Generic.Tests.cs(859,0): at System.Collections.Tests.HashSet_Generic_Tests`1.ComparerSerialization()
             at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
          D:\projects\runtime\src\libraries\System.Private.CoreLib\src\System\Reflection\MethodBaseInvoker.cs(57,0): at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
      System.Collections.Tests.HashSet_Generic_Tests_int_With_Comparer_WrapStructural_SimpleInt.ComparerSerialization [FAIL]
        Assert.IsType() Failure: Value is not the exact type
        Expected: typeof(System.Collections.Generic.StringEqualityComparer)
        Actual:   typeof(System.Collections.Generic.GenericEqualityComparer<string>)
        Stack Trace:
          D:\projects\runtime\src\libraries\System.Collections\tests\Generic\HashSet\HashSet.Generic.Tests.cs(891,0): at System.Collections.Tests.HashSet_Generic_Tests`1.<ComparerSerialization>g__TestComparerSerialization|64_0[TCompare
  d](IEqualityComparer`1 equalityComparer, String internalTypeName)
          D:\projects\runtime\src\libraries\System.Collections\tests\Generic\HashSet\HashSet.Generic.Tests.cs(859,0): at System.Collections.Tests.HashSet_Generic_Tests`1.ComparerSerialization()
             at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
          D:\projects\runtime\src\libraries\System.Private.CoreLib\src\System\Reflection\MethodBaseInvoker.cs(57,0): at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
      System.Collections.Tests.HashSet_Generic_Tests_int_With_Comparer_WrapStructural_Int.ComparerSerialization [FAIL]
        Assert.IsType() Failure: Value is not the exact type
        Expected: typeof(System.Collections.Generic.StringEqualityComparer)
        Actual:   typeof(System.Collections.Generic.GenericEqualityComparer<string>)
        Stack Trace:
          D:\projects\runtime\src\libraries\System.Collections\tests\Generic\HashSet\HashSet.Generic.Tests.cs(891,0): at System.Collections.Tests.HashSet_Generic_Tests`1.<ComparerSerialization>g__TestComparerSerialization|64_0[TCompare
  d](IEqualityComparer`1 equalityComparer, String internalTypeName)
          D:\projects\runtime\src\libraries\System.Collections\tests\Generic\HashSet\HashSet.Generic.Tests.cs(859,0): at System.Collections.Tests.HashSet_Generic_Tests`1.ComparerSerialization()
             at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
          D:\projects\runtime\src\libraries\System.Private.CoreLib\src\System\Reflection\MethodBaseInvoker.cs(57,0): at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
    Finished:    System.Collections.Tests

bartonjs · 2024-08-26T16:27:05Z

There are 5 System.Collections.Tests tests that fail due to a product change. I think you have mentioned it at some point, but I just want to make sure that we are aware of that change, it’s documented and expected.

If I did, I have no recollection of it. The BF compat package is just BF. Nothing should be different between using it in 9 and using BF in 8.

ericstj · 2024-08-26T16:29:22Z

How adding such project reference is going to affect the build time for those configurations? Should we somehow make it conditional?

It shouldn't add any time to a root level build. This project already builds and it will continue to only build once - MSBuild will get the cached value. As for leaf builds of individual projects - it will increase the time, however if folks want to avoid that they can always pass --no-dependencies to skip project references.

I wonder if we really want to have all these binary-formatter tests spread across the repo. Maybe we should consolidate them to limit the number of tests that use the OOB?

bartonjs · 2024-08-26T16:30:39Z

src/libraries/Common/tests/TestUtilities/System/PlatformDetection.cs

-            // Assembly versions beyond 8.1 are the fully functional version from NuGet.
-            // Assembly versions before 8.1 probably won't be encountered, since that's the past.
-
-            if (assemblyVersion.Major == 8 && assemblyVersion.Minor == 1)


I don't understand why you're taking the assembly version logic out.

I don't understand why you're taking the assembly version logic out.

Because it has hidden the fact that 19 projects were using the "throwing" version of BF (the tests were always skipped so any test failures would go unnoticed). With this change such tests are going to throw if they reference the wrong version.

…t is mapped to StringEqualityComparer, but serialized as GenericEqualityComparer<string>

adamsitnik · 2024-08-29T14:51:30Z

@bartonjs you were right, the failures were caused by changes from #104202. I've fixed them, the tests should now pass.

The best explanation can be found here:

runtime/src/libraries/System.Private.CoreLib/src/System/Collections/Generic/EqualityComparer.cs

Lines 277 to 295 in c05e9d1

    
           // This type is added as an internal implementation detail in .NET 9. Even though as of .NET 9 BinaryFormatter has been 
        
           // deprecated, for back compat we still need to support serializing this type, especially when EqualityComparer<string>.Default 
        
           // is used as part of a collection, like Dictionary<string, TValue>. 
        
           // 
        
           // BinaryFormatter treats types in the core library as being special, in that it doesn't include the assembly as part of the 
        
           // serialized data, and then on deserialization it assumes the type is in mscorlib. We could make the type public and type forward 
        
           // it from the mscorlib shim, which would enable roundtripping on .NET 9+, but because this type doesn't exist downlevel, it would 
        
           // break serializing on .NET 9+ and deserializing downlevel. Therefore, we need to serialize as something that exists downlevel. 
        
           // We could serialize as OrdinalComparer, which does exist downlevel, and which has the nice property that it also implements 
        
           // IAlternateEqualityComparer<ReadOnlySpan<char>, string>, which means serializing an instance on .NET 9+ and deserializing it 
        
           // on .NET 9+ would continue to support span-based lookups. However, OrdinalComparer is not an EqualityComparer<string>, which 
        
           // means the type's public ancestry would not be retained, which could lead to strange casting-related errors, including downlevel. 
        
           // Instead, we can serialize as a GenericEqualityComparer<string>. This exists downlevel and also derives from EqualityComparer<string>, 
        
           // but doesn't implement IAlternateEqualityComparer<ReadOnlySpan<char>, string>. This means that upon deserializing on .NET 9+, 
        
           // the comparer loses its ability to handle span-based lookups. As BinaryFormatter is deprecated on .NET 9+, this is a readonable tradeoff. 
        
           info.SetType(typeof(GenericEqualityComparer<string>));

adamsitnik · 2024-08-29T14:52:26Z

I wonder if we really want to have all these binary-formatter tests spread across the repo. Maybe we should consolidate them to limit the number of tests that use the OOB?

I like this idea, but due to time constraints I would prefer to do it after we ship 9. Is that acceptable?

adamsitnik · 2024-08-30T20:02:46Z

@ericstj @bartonjs the PR is ready for review, PTAL

…otnet#106858) * respect AppContext switch (which is currently enabled for all projects in the root Directory.Build.props file) * add project reference to all test projects that need working BF (and were being skipped for a while) * adjust to changes from dotnet#104202: EqualityComparer<string>.Default is mapped to StringEqualityComparer, but serialized as GenericEqualityComparer<string>

…106858) * respect AppContext switch (which is currently enabled for all projects in the root Directory.Build.props file) * add project reference to all test projects that need working BF (and were being skipped for a while) * adjust to changes from #104202: EqualityComparer<string>.Default is mapped to StringEqualityComparer, but serialized as GenericEqualityComparer<string>

* Remove package references from library tests (#106737) * Remove package references from library tests These tests should be referencing the product assemblies so that they test latest and not old bits. * Reference the OOB version of SRSF and make sure it's copied * BinaryFormatter tests should be skipped only on AOT, WASM and Mobile (#106858) * respect AppContext switch (which is currently enabled for all projects in the root Directory.Build.props file) * add project reference to all test projects that need working BF (and were being skipped for a while) * adjust to changes from #104202: EqualityComparer<string>.Default is mapped to StringEqualityComparer, but serialized as GenericEqualityComparer<string> * Don't use WeakReferences in the round trip test, as the target may get freed in the meantime, fixes #104905 (#106967) * Enable more BinaryFormatter tests (#107408) * enable the BinaryFormatter tests in System.Runtime.Serialization.Formatters.Tests * add new test project, where the flag is disabled and it runs only 3 tests in total that ensure that * The SerializationGuard is no longer activated since BF was moved to the OOB package, the tests need to reflect that. * Disable binary formatter tests when DotNetBuildSourceOnly. (#107549) * [mono][tvos] Do not treat assembly.pdb/xml files as native files to bundle when AOTing on Helix (#107079) * Do not treat assembly.pdb/xml files as native files to bundle * Bundle satellite assemblies as well * [mono][ci] Include PDBs from runtime pack when building on Helix if required (#107348) --------- Co-authored-by: Eric StJohn <ericstj@microsoft.com> Co-authored-by: Tom Deseyn <tom.deseyn@gmail.com> Co-authored-by: Ivan Povazan <55002338+ivanpovazan@users.noreply.github.com>

…107903) * Remove package references from library tests (#106737) * Remove package references from library tests These tests should be referencing the product assemblies so that they test latest and not old bits. * Reference the OOB version of SRSF and make sure it's copied * BinaryFormatter tests should be skipped only on AOT, WASM and Mobile (#106858) * respect AppContext switch (which is currently enabled for all projects in the root Directory.Build.props file) * add project reference to all test projects that need working BF (and were being skipped for a while) * adjust to changes from #104202: EqualityComparer<string>.Default is mapped to StringEqualityComparer, but serialized as GenericEqualityComparer<string> * Don't use WeakReferences in the round trip test, as the target may get freed in the meantime, fixes #104905 (#106967) * Enable more BinaryFormatter tests (#107408) * enable the BinaryFormatter tests in System.Runtime.Serialization.Formatters.Tests * add new test project, where the flag is disabled and it runs only 3 tests in total that ensure that * The SerializationGuard is no longer activated since BF was moved to the OOB package, the tests need to reflect that. * Disable binary formatter tests when DotNetBuildSourceOnly. (#107549) * [mono][tvos] Do not treat assembly.pdb/xml files as native files to bundle when AOTing on Helix (#107079) * Do not treat assembly.pdb/xml files as native files to bundle * Bundle satellite assemblies as well * [mono][ci] Include PDBs from runtime pack when building on Helix if required (#107348) --------- Co-authored-by: Eric StJohn <ericstj@microsoft.com> Co-authored-by: Adam Sitnik <adam.sitnik@gmail.com> Co-authored-by: Tom Deseyn <tom.deseyn@gmail.com> Co-authored-by: Ivan Povazan <55002338+ivanpovazan@users.noreply.github.com>

…otnet#106858) * respect AppContext switch (which is currently enabled for all projects in the root Directory.Build.props file) * add project reference to all test projects that need working BF (and were being skipped for a while) * adjust to changes from dotnet#104202: EqualityComparer<string>.Default is mapped to StringEqualityComparer, but serialized as GenericEqualityComparer<string>

IsBinaryFormatterSupported should return false only on platforms wher…

19bbf2f

…e it's not supported by design

adamsitnik requested review from ericstj and bartonjs August 23, 2024 06:43

dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Aug 23, 2024

dotnet-policy-service bot assigned adamsitnik Aug 23, 2024

adamsitnik mentioned this pull request Aug 23, 2024

Remove package references from library tests #106737

Merged

adamsitnik added area-System.Runtime binaryformatter-migration Issues related to the removal of BinaryFormatter and migrations away from it and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Aug 23, 2024

adamsitnik added the test-enhancement Improvements of test source code label Aug 23, 2024

build-analysis bot mentioned this pull request Aug 23, 2024

Checkout failure: "Git fetch failed with exit code 128" dotnet/arcade#9009

Open

2 tasks

adamsitnik added 2 commits August 26, 2024 11:26

respect AppContext switch (which is currently enabled for all project…

650c3d6

…s in the root Directory.Build.props file)

add project reference to all test projects that need working BF (and …

1a95f34

…were being skipped for a while)

build-analysis bot mentioned this pull request Aug 26, 2024

SIGKILL (OOM?) while running LibraryImportGenerator.Tests w/o actionable log messages or artifacts dotnet/dnceng#2496

Open

3 tasks

bartonjs reviewed Aug 26, 2024

View reviewed changes

adjust to changes from dotnet#104202: EqualityComparer<string>.Defaul…

2bfec6b

…t is mapped to StringEqualityComparer, but serialized as GenericEqualityComparer<string>

adamsitnik requested a review from bartonjs August 29, 2024 14:52

build-analysis bot mentioned this pull request Aug 29, 2024

[mono][tvos] Libraries tests failed to build on Helix due to missing System.Runtime.Serialization.Formatters.* files #107029

Closed

bartonjs approved these changes Aug 30, 2024

View reviewed changes

adamsitnik merged commit 4d501d9 into dotnet:main Sep 2, 2024
81 of 85 checks passed

matouskozak mentioned this pull request Sep 3, 2024

System.Runtime.Serialization.Formatters.pdb not found #107285

Closed

akoeplinger mentioned this pull request Sep 3, 2024

Target .NET 10 / net10.0 #106599

Draft

tmds mentioned this pull request Sep 9, 2024

BinaryFormatter tests are no longer skipped when ran against source-built runtime #107534

Closed

adamsitnik mentioned this pull request Sep 9, 2024

[release/9.0] BinaryFormatter tests improvements #107540

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BinaryFormatter tests should be skipped only on AOT, WASM and Mobile #106858

BinaryFormatter tests should be skipped only on AOT, WASM and Mobile #106858

adamsitnik commented Aug 23, 2024

dotnet-policy-service bot commented Aug 23, 2024

adamsitnik commented Aug 26, 2024

bartonjs commented Aug 26, 2024

ericstj commented Aug 26, 2024

bartonjs Aug 26, 2024

adamsitnik Aug 27, 2024

adamsitnik commented Aug 29, 2024

adamsitnik commented Aug 29, 2024

adamsitnik commented Aug 30, 2024

BinaryFormatter tests should be skipped only on AOT, WASM and Mobile #106858

BinaryFormatter tests should be skipped only on AOT, WASM and Mobile #106858

Conversation

adamsitnik commented Aug 23, 2024

dotnet-policy-service bot commented Aug 23, 2024

adamsitnik commented Aug 26, 2024

bartonjs commented Aug 26, 2024

ericstj commented Aug 26, 2024

bartonjs Aug 26, 2024

Choose a reason for hiding this comment

adamsitnik Aug 27, 2024

Choose a reason for hiding this comment

adamsitnik commented Aug 29, 2024

adamsitnik commented Aug 29, 2024

adamsitnik commented Aug 30, 2024