Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Assertion failed: OpenSsl error queue is not empty #57722

Closed
karelz opened this issue Aug 19, 2021 · 6 comments · Fixed by #64256
Closed

Assertion failed: OpenSsl error queue is not empty #57722

karelz opened this issue Aug 19, 2021 · 6 comments · Fixed by #64256
Assignees
Labels
area-System.Net.Security os-linux Linux OS (any supported distro) test-run-core Test failures in .NET Core test runs
Milestone

Comments

@karelz
Copy link
Member

karelz commented Aug 19, 2021

CoreCLR run of System.Net.Security.Tests:

Failures 7/30-8/20 (incl. PRs):

Day Run OS Runtime Notes
7/30 PR #56561 Ubuntu.1604.Amd64.Open Mono Log
7/30 PR #56628 Ubuntu.1804.ArmArch.Open - arm64 Mono
8/2 PR #56720 Alpine.312.Amd64.Open CoreCLR
8/3 PR #56568 Fedora.34.Amd64.Open CoreCLR
8/3 PR #56793 Ubuntu.1604.Amd64.Open CoreCLR
8/6 PR #56714 SLES.15.Amd64.Open Mono
8/6 PR #56911 SLES.15.Amd64.Open Mono
8/9 PR #56173 SLES.15.Amd64.Open CoreCLR
8/10 PR #57162 Alpine.312.Amd64.Open CoreCLR
8/12 PR #57113 Ubuntu.1804.Amd64.Open Mono
8/17 PR #57541 Ubuntu.1604.Amd64.Open CoreCLR Log
8/19 PR #45690 Debian.10.Amd64.Open Mono
8/19 PR #56501 Fedora.34.Amd64.Open CoreCLR
8/30 PR #58314 Ubuntu.1804.ArmArch.Open - arm64 Mono

Failure Mono - Encrypt:

Process terminated due to "OpenSsl error queue is not empty, run: 'openssl errstr 1407742E' for original error.
   at System.Diagnostics.DebugProvider.Fail(String message, String detailMessage) in /_/src/libraries/System.Private.CoreLib/src/System/Diagnostics/DebugProvider.cs:line 22
   at System.Diagnostics.Debug.Fail(String message, String detailMessage) in /_/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Debug.cs:line 134
   at System.Diagnostics.Debug.Assert(Boolean condition, String message, String detailMessage) in /_/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Debug.cs:line 98
   at System.Diagnostics.Debug.Assert(Boolean condition, String message) in /_/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Debug.cs:line 87
   at System.Diagnostics.Debug.Assert(Boolean condition, AssertInterpolatedStringHandler message) in /_/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Debug.cs:line 91
   at Interop.OpenSsl.Encrypt(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& output, SslErrorCode& errorCode) in /_/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs:line 312
   at System.Net.Security.SslStreamPal.EncryptMessage(SafeDeleteSslContext securityContext, ReadOnlyMemory`1 input, Int32 headerSize, Int32 trailerSize, Byte[]& output, Int32& resultSize) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.Unix.cs:line 49
   at System.Net.Security.SecureChannel.Encrypt(ReadOnlyMemory`1 buffer, Byte[]& output, Int32& resultSize) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SecureChannel.cs:line 886
   at System.Net.Security.SslStream.EncryptData(ReadOnlyMemory`1 buffer, Byte[]& outBuffer, Int32& outSize) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 184
   at System.Net.Security.SslStream.WriteSingleChunk[AsyncReadWriteAdapter](AsyncReadWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 692
   at System.Net.Security.SslStream.<WriteAsyncInternal>d__189`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext() in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 1151
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[<WriteAsyncInternal>d__189`1](<WriteAsyncInternal>d__189`1& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 38
   at System.Net.Security.SslStream.WriteAsyncInternal[AsyncReadWriteAdapter](AsyncReadWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer) in System.Net.Security.dll:token 0x6000344+0x0
   at System.Net.Security.SslStream.WriteAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs:line 818
   at System.IO.Tests.ConnectedStreamConformanceTests.<>c__DisplayClass31_0.<<Read_Eof_Returns0>b__0>d.MoveNext() in /_/src/libraries/Common/tests/StreamConformanceTests/System/IO/StreamConformanceTests.cs:line 1849
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[<<Read_Eof_Returns0>b__0>d](<<Read_Eof_Returns0>b__0>d& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 38
   at System.IO.Tests.ConnectedStreamConformanceTests.<>c__DisplayClass31_0.<Read_Eof_Returns0>b__0() in StreamConformanceTests.dll:token 0x6000334+0x0
   at System.Threading.Tasks.Task`1[[System.Threading.Tasks.Task, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].InnerInvoke() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Future.cs:line 503
   at System.Threading.Tasks.Task.<>c.<.cctor>b__271_0(Object obj) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2369
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 268
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2331
   at System.Threading.Tasks.Task.ExecuteEntryUnsafe(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2265
   at System.Threading.Tasks.Task.ExecuteFromThreadPool(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2256
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 708
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs:line 63
   at System.Threading.Thread.StartCallback() in /_/src/mono/System.Private.CoreLib/src/System/Threading/Thread.Mono.cs:line 236"

Failure CoreCLR - Decrypt:

Process terminated. Assertion failed.
OpenSsl error queue is not empty, run: 'openssl errstr 1407742E' for original error.
   at Interop.OpenSsl.Encrypt(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& output, SslErrorCode& errorCode) in /_/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs:line 312
   at System.Net.Security.SslStreamPal.EncryptMessage(SafeDeleteSslContext securityContext, ReadOnlyMemory`1 input, Int32 headerSize, Int32 trailerSize, Byte[]& output, Int32& resultSize) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.Unix.cs:line 49
   at System.Net.Security.SecureChannel.Encrypt(ReadOnlyMemory`1 buffer, Byte[]& output, Int32& resultSize) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SecureChannel.cs:line 886
   at System.Net.Security.SslStream.EncryptData(ReadOnlyMemory`1 buffer, Byte[]& outBuffer, Int32& outSize) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 184
   at System.Net.Security.SslStream.WriteSingleChunk[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 693
   at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 1152
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 38
   at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer)
   at System.Net.Security.SslStream.WriteAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs:line 818
   at System.IO.Tests.ConnectedStreamConformanceTests.ReadAsync_ContinuesOnCurrentTaskSchedulerIfDesired(Boolean flowExecutionContext, Nullable`1 continueOnCapturedContext) in /_/src/libraries/Common/tests/StreamConformanceTests/System/IO/StreamConformanceTests.cs:line 2145
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 268
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 328
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecuteFromThreadPool(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 299
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 708
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs:line 63
   at System.Threading.Thread.StartCallback() in /_/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 105

Failure CoreCLR - Decrypt:

Process terminated. Assertion failed.
OpenSsl error queue is not empty, run: 'openssl errstr 1409442E' for original error.
   at Interop.OpenSsl.Decrypt(SafeSslHandle context, Span`1 buffer, SslErrorCode& errorCode) in /_/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs:line 366
   at System.Net.Security.SslStreamPal.DecryptMessage(SafeDeleteSslContext securityContext, Span`1 buffer, Int32& offset, Int32& count) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.Unix.cs:line 67
   at System.Net.Security.SecureChannel.Decrypt(Span`1 buffer, Int32& outputOffset, Int32& outputCount) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SecureChannel.cs:line 908
   at System.Net.Security.SslStream.DecryptData(Int32 frameSize) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 874
   at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 972
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 183
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 324
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 795
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 3368
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Future.cs:line 400
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 441
   at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](TIOAdapter adapter) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 859
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 183
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 324
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 795
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 3368
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Future.cs:line 400
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 441
   at System.IO.StreamBuffer.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/Common/src/System/Net/StreamBuffer.cs:line 264
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 268
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 328
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 708
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs:line 63
   at System.Threading.Thread.StartCallback() in /_/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 105
@karelz karelz added area-System.Net.Security test-run-core Test failures in .NET Core test runs labels Aug 19, 2021
@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Aug 19, 2021
@ghost
Copy link

ghost commented Aug 19, 2021

Tagging subscribers to this area: @dotnet/ncl, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

CoreCLR run of System.Net.Security.Tests:

Day Run OS Notes
8/17 PR #57541 Ubuntu.1604.Amd64.Open Log

Failure:

Process terminated. Assertion failed.
OpenSsl error queue is not empty, run: 'openssl errstr 1407742E' for original error.
   at Interop.OpenSsl.Encrypt(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& output, SslErrorCode& errorCode) in /_/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs:line 312
   at System.Net.Security.SslStreamPal.EncryptMessage(SafeDeleteSslContext securityContext, ReadOnlyMemory`1 input, Int32 headerSize, Int32 trailerSize, Byte[]& output, Int32& resultSize) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.Unix.cs:line 49
   at System.Net.Security.SecureChannel.Encrypt(ReadOnlyMemory`1 buffer, Byte[]& output, Int32& resultSize) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SecureChannel.cs:line 886
   at System.Net.Security.SslStream.EncryptData(ReadOnlyMemory`1 buffer, Byte[]& outBuffer, Int32& outSize) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 184
   at System.Net.Security.SslStream.WriteSingleChunk[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 693
   at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 1152
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 38
   at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer)
   at System.Net.Security.SslStream.WriteAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs:line 818
   at System.IO.Tests.ConnectedStreamConformanceTests.ReadAsync_ContinuesOnCurrentTaskSchedulerIfDesired(Boolean flowExecutionContext, Nullable`1 continueOnCapturedContext) in /_/src/libraries/Common/tests/StreamConformanceTests/System/IO/StreamConformanceTests.cs:line 2145
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 268
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 328
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecuteFromThreadPool(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 299
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 708
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs:line 63
   at System.Threading.Thread.StartCallback() in /_/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 105
Author: karelz
Assignees: -
Labels:

area-System.Net.Security, test-run-core

Milestone: -

@karelz karelz added the os-linux Linux OS (any supported distro) label Aug 19, 2021
@karelz karelz modified the milestones: 6.0.0, 7.0.0 Aug 19, 2021
@jeffschwMSFT jeffschwMSFT removed the untriaged New issue has not been triaged by the area owner label Aug 19, 2021
@wfurt
Copy link
Member

wfurt commented Aug 24, 2021

error:1407742E:SSL routines:func(119):tlsv1 alert protocol version

@jkotas
Copy link
Member

jkotas commented Jan 20, 2022

Hit by #64057

@wfurt
Copy link
Member

wfurt commented Jan 21, 2022

I hit it today few times while working on something else with debug version of OpenSSL. I'll take a look.

@wfurt wfurt self-assigned this Jan 21, 2022
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Jan 25, 2022
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Feb 4, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Mar 6, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-System.Net.Security os-linux Linux OS (any supported distro) test-run-core Test failures in .NET Core test runs
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants