System.Net.Security.Tests failed with System.IO.IOException

[VincentBu]

Run: runtime-coreclr libraries-jitstress 20210505.1

Failed tests:

net6.0-windows-Release-arm64-CoreCLR_checked-no_tiered_compilation-Windows.10.Arm64v8.Open
 -System.Net.Security.Tests.ClientAsyncAuthenticateTest.ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success(serverProtocol: Ssl3)
 -System.Net.Security.Tests.ServerAsyncAuthenticateTest.ServerAsyncAuthenticate_EachSupportedProtocol_Success(protocol: Ssl3)

Error message:

System.IO.IOException :  Received an unexpected EOF or 0 bytes from the transport stream.


Stack trace
   at System.Net.Security.SslStream.<FillHandshakeBufferAsync>g__InternalFillHandshakeBufferAsync|181_0[TIOAdapter](TIOAdapter adap, ValueTask`1 task, Int32 minSize) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 953
   at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 456
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 358
   at System.Net.Security.Tests.ClientAsyncAuthenticateTest.ClientAsyncSslHelper(EncryptionPolicy encryptionPolicy, SslProtocols clientSslProtocols, SslProtocols serverSslProtocols, RemoteCertificateValidationCallback certificateCallback) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/ClientAsyncAuthenticateTest.cs:line 155
   at System.Net.Security.Tests.ClientAsyncAuthenticateTest.ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success(SslProtocols serverProtocol) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/ClientAsyncAuthenticateTest.cs:line 102
--- End of stack trace from previous location ---

[ghost]

Tagging subscribers to this area: @dotnet/ncl, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Run: runtime-coreclr libraries-jitstress 20210505.1

Failed tests:

net6.0-windows-Release-arm64-CoreCLR_checked-no_tiered_compilation-Windows.10.Arm64v8.Open
 -System.Net.Security.Tests.ClientAsyncAuthenticateTest.ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success(serverProtocol: Ssl3)
 -System.Net.Security.Tests.ServerAsyncAuthenticateTest.ServerAsyncAuthenticate_EachSupportedProtocol_Success(protocol: Ssl3)

Error message:

System.IO.IOException :  Received an unexpected EOF or 0 bytes from the transport stream.


Stack trace
   at System.Net.Security.SslStream.<FillHandshakeBufferAsync>g__InternalFillHandshakeBufferAsync|181_0[TIOAdapter](TIOAdapter adap, ValueTask`1 task, Int32 minSize) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 953
   at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 456
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 358
   at System.Net.Security.Tests.ClientAsyncAuthenticateTest.ClientAsyncSslHelper(EncryptionPolicy encryptionPolicy, SslProtocols clientSslProtocols, SslProtocols serverSslProtocols, RemoteCertificateValidationCallback certificateCallback) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/ClientAsyncAuthenticateTest.cs:line 155
   at System.Net.Security.Tests.ClientAsyncAuthenticateTest.ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success(SslProtocols serverProtocol) in /_/src/libraries/System.Net.Security/tests/FunctionalTests/ClientAsyncAuthenticateTest.cs:line 102
--- End of stack trace from previous location ---

Author:	VincentBu
Assignees:	-
Labels:	JitStress, arch-arm64, area-System.Net.Security, os-windows
Milestone:	-

[karelz]

Kusto -- methods to query:

  or Method == 'ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success'
  or Method == 'ServerAsyncAuthenticate_EachSupportedProtocol_Success'

Failures 3/18-9/2 (incl. PRs):

Date	Build	OS	Test
5/5	no_tiered_compilation	Windows.10.Arm64v8.Open	ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success ServerAsyncAuthenticate_EachSupportedProtocol_Success
5/19	jitstress2_tiered	Windows.10.Arm64v8.Open	ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success ServerAsyncAuthenticate_EachSupportedProtocol_Success
5/22	jitstress2_jitstressregs0x80	Windows.10.Arm64v8.Open	ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success ServerAsyncAuthenticate_EachSupportedProtocol_Success
6/9	jitstress2_tiered	Windows.10.Arm64v8.Open	ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success ServerAsyncAuthenticate_EachSupportedProtocol_Success
6/13	zapdisable	Windows.10.Arm64v8.Open	ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success ServerAsyncAuthenticate_EachSupportedProtocol_Success
6/14	no_tiered_compilation	Windows.10.Arm64v8.Open	ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success ServerAsyncAuthenticate_EachSupportedProtocol_Success
6/15	zapdisable	Windows.10.Arm64v8.Open	ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success ServerAsyncAuthenticate_EachSupportedProtocol_Success
6/22	jitstress2_tiered	Windows.10.Arm64v8.Open	ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success ServerAsyncAuthenticate_EachSupportedProtocol_Success
6/26	jitstress2_jitstressregs4	Windows.10.Arm64v8.Open	ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success ServerAsyncAuthenticate_EachSupportedProtocol_Success
6/30	Official run	Windows.Server.Core.1909.Amd64.Open	ServerAsyncAuthenticate_EachSupportedProtocol_Success
7/4	tailcallstress	Windows.10.Arm64v8.Open	ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success ServerAsyncAuthenticate_EachSupportedProtocol_Success
7/5	jitstress1	Windows.10.Arm64v8.Open	ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success ServerAsyncAuthenticate_EachSupportedProtocol_Success
7/8	PR #55298	Windows.10.Amd64.Client21H1.Open	ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success ServerAsyncAuthenticate_EachSupportedProtocol_Success
7/11	jitstressregs0x1000	Windows.10.Arm64v8.Open	ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success ServerAsyncAuthenticate_EachSupportedProtocol_Success
7/21	no_tiered_compilation	Windows.10.Arm64v8.Open	ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success ServerAsyncAuthenticate_EachSupportedProtocol_Success
7/23	PR #55298	Windows.10.Amd64.Client21H1.Open	ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success ServerAsyncAuthenticate_EachSupportedProtocol_Success
7/31	tailcallstress	Windows.10.Arm64v8.Open	ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success ServerAsyncAuthenticate_EachSupportedProtocol_Success
8/1	jitstressregs0x80	Windows.10.Arm64v8.Open	ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success ServerAsyncAuthenticate_EachSupportedProtocol_Success
8/4	tailcallstress	Windows.10.Arm64v8.Open	ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success ServerAsyncAuthenticate_EachSupportedProtocol_Success
8/13	PR #57282 - jitstress1_tiered	Windows.10.Arm64v8.Open	ServerAsyncAuthenticate_EachSupportedProtocol_Success
8/15	jitstress2	Windows.10.Arm64v8.Open	ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success ServerAsyncAuthenticate_EachSupportedProtocol_Success
8/15	defaultpgo	Windows.10.Arm64v8.Open	ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success ServerAsyncAuthenticate_EachSupportedProtocol_Success
8/15	dynamicpgo	Windows.10.Arm64v8.Open	ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success ServerAsyncAuthenticate_EachSupportedProtocol_Success
8/15	nopgo	Windows.10.Arm64v8.Open	ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success ServerAsyncAuthenticate_EachSupportedProtocol_Success
9/1	jitstress1_tiered	Windows.10.Arm64v8.Open	ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success ServerAsyncAuthenticate_EachSupportedProtocol_Success
9/2	jitstress2	Windows.10.Arm64v8.Open	ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success ServerAsyncAuthenticate_EachSupportedProtocol_Success
9/4	jitstress1_tiered	Windows.10.Arm64v8.Open	ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success ServerAsyncAuthenticate_EachSupportedProtocol_Success
9/6	jitstress2_tiered	Windows.10.Arm64v8.Open	ClientAsyncAuthenticate_IndividualServerVsAllClientSupportedProtocols_Success ServerAsyncAuthenticate_EachSupportedProtocol_Success

It is suspicious that the tests fail only with JitStress and all tests at once, also always together with test failures in #52361 -- perhaps some kind of environmental problem?

[danmoseley]

@BruceForstall could you help hypothesize why this only fails for jitstress? Is this a particularly slow form of jitstress?

[BruceForstall]

@BruceForstall could you help hypothesize why this only fails for jitstress? Is this a particularly slow form of jitstress?

Based on the table above, the test fails in many different types of jitstress. Some are slower (like zapdisable), but many should be much faster than normal, e.g., "no_tiered_compilation" which just says "turn on the JIT optimizer; don't allow tiering". Possibly some timing issue is exacerbated by the code running faster (due to the JIT optimizing the code)?

Note that one problem with most libraries tests is that the default run style is with tiered compilation enabled. That means that the test runs with no JIT optimization until it runs enough times to be "tiered up" / optimized. But most test code never reaches that level. However, the jitstress modes all disable tiering, forcing JIT optimization.

Has anyone tried reproducing these locally?

[danmoseley]

Again coincident with #52361. @karelz probably one could dupe against the other.

[karelz]

@danmoseley the exceptions are different, so keeping 2 different issues isn't bad. No need to dupe them.

[wfurt]

I think this duplicate of #52361. Depending on timing we can surface the server exception but if the client wins, we would see connection close as the server aborted and the exception is still to be processed.

await TestConfiguration.WhenAllOrAnyFailedWithTimeout(t1, t2);