Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI failure: system.security.cryptography.openssl.tests missing libssl #73210

Closed
runfoapp bot opened this issue Aug 2, 2022 · 8 comments
Closed

CI failure: system.security.cryptography.openssl.tests missing libssl #73210

runfoapp bot opened this issue Aug 2, 2022 · 8 comments
Labels
area-System.Security blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' tracking-external-issue The issue is caused by external problem (e.g. OS) - nothing we can do to fix it directly
Milestone
7.0.0

Comments

@runfoapp
Copy link

runfoapp bot commented Aug 2, 2022 •
edited
Loading

Runfo shows a bunch of failures, but looking at the first few none of them looked similar to the missing libssl error shown here.
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-d48929e09acb474baa/System.Security.Cryptography.OpenSsl.Tests/1/console.a5cb0446.log?%3Fhelixlogtype%3Dresult

/private/tmp/helix/working/B16B09B5/w/AB2909A1/e /private/tmp/helix/working/B16B09B5/w/AB2909A1/e
  Discovering: System.Security.Cryptography.OpenSsl.Tests (method display = ClassAndMethod, method display options = None)
No usable version of libssl was found
./RunTests.sh: line 168: 59278 Abort trap: 6           "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Security.Cryptography.OpenSsl.Tests.runtimeconfig.json --depsfile System.Security.Cryptography.OpenSsl.Tests.deps.json xunit.console.dll System.Security.Cryptography.OpenSsl.Tests.dll -xml

Runfo Tracking Issue: system.security.cryptography.openssl.tests missing libssl

Build Definition Kind Run Name Console Core Dump Test Results Run Client
1927756 runtime PR 73520 net7.0-Linux-Debug-arm64-Mono_release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20220427172132-97d8652 console.log core dump runclient.py
1916354 runtime PR 73185 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1916354 runtime PR 73185 net7.0-Linux-Release-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open console.log core dump runclient.py
1916354 runtime PR 73185 net7.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.314.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm64v8-20210910135810-8a6f4f3 console.log core dump runclient.py
1916354 runtime PR 73185 net7.0-Linux-Debug-arm64-CoreCLR_checked-(Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652 console.log core dump runclient.py
1916354 runtime PR 73185 net7.0-Linux-Debug-x64-CoreCLR_checked-(Alpine.314.Amd64.Open)Ubuntu.1804.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-amd64-20210910135833-1848e19 console.log core dump runclient.py
1915309 runtime PR 72934 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1915051 runtime PR 49576 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1915051 runtime PR 49576 net7.0-Linux-Release-arm-CoreCLR_checked-(Alpine.314.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3 console.log core dump runclient.py
1915051 runtime PR 49576 net7.0-Linux-Debug-arm64-CoreCLR_checked-(Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652 console.log core dump runclient.py
1915051 runtime PR 49576 net7.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.314.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm64v8-20210910135810-8a6f4f3 console.log core dump runclient.py
1915051 runtime PR 49576 net7.0-Linux-Release-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open console.log core dump runclient.py
1915051 runtime PR 49576 net7.0-Linux-Debug-x64-CoreCLR_checked-(Alpine.314.Amd64.Open)Ubuntu.1804.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-amd64-20210910135833-1848e19 console.log core dump runclient.py
1914778 runtime PR 49576 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1914778 runtime PR 49576 net7.0-Linux-Release-arm-CoreCLR_checked-(Alpine.314.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3 console.log core dump runclient.py
1914778 runtime PR 49576 net7.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.314.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm64v8-20210910135810-8a6f4f3 console.log core dump runclient.py
1914778 runtime PR 49576 net7.0-Linux-Debug-arm64-CoreCLR_checked-(Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652 console.log core dump runclient.py
1914778 runtime PR 49576 net7.0-Linux-Release-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open console.log core dump runclient.py
1914778 runtime PR 49576 net7.0-Linux-Debug-x64-CoreCLR_checked-(Alpine.314.Amd64.Open)Ubuntu.1804.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-amd64-20210910135833-1848e19 console.log core dump runclient.py
1914601 runtime Rolling net7.0-OSX-Release-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1914164 runtime PR 73120 net7.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.314.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm64v8-20210910135810-8a6f4f3 console.log core dump runclient.py
1914164 runtime PR 73120 net7.0-Linux-Debug-arm64-CoreCLR_checked-(Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652 console.log core dump runclient.py
1914164 runtime PR 73120 net7.0-Linux-Release-arm-CoreCLR_checked-(Alpine.314.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3 console.log core dump runclient.py
1914164 runtime PR 73120 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1914164 runtime PR 73120 net7.0-Linux-Release-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open console.log core dump runclient.py
1914164 runtime PR 73120 net7.0-Linux-Debug-x64-CoreCLR_checked-(Alpine.314.Amd64.Open)Ubuntu.1804.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-amd64-20210910135833-1848e19 console.log core dump runclient.py
1913265 runtime PR 73105 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1913236 runtime PR 73095 net7.0-Linux-Debug-arm64-CoreCLR_checked-(Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652 console.log runclient.py
1913236 runtime PR 73095 net7.0-Linux-Debug-arm64-Mono_release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20220427172132-97d8652 console.log runclient.py
1913236 runtime PR 73095 net7.0-Linux-Debug-x64-Mono_release-(Debian.10.Amd64.Open)Ubuntu.1804.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-10-helix-amd64-20210304164434-56c6673 console.log runclient.py
1913236 runtime PR 73095 net7.0-Linux-Release-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open console.log runclient.py
1913236 runtime PR 73095 net7.0-Linux-Debug-x64-Mono_release-Ubuntu.1804.Amd64.Open console.log runclient.py
1913236 runtime PR 73095 net7.0-Linux-Release-arm-CoreCLR_checked-(Alpine.314.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm32v7-20210910135806-8a6f4f3 console.log runclient.py
1913236 runtime PR 73095 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1913236 runtime PR 73095 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1913236 runtime PR 73095 net7.0-Linux-Debug-x64-CoreCLR_release-(Centos.7.Amd64.Open)Ubuntu.1604.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7-mlnet-helix-20220601183719-dde38af console.log runclient.py
1913236 runtime PR 73095 net7.0-Linux-Debug-x64-CoreCLR_release-RedHat.7.Amd64.Open console.log runclient.py
1913236 runtime PR 73095 net7.0-Linux-Debug-x64-CoreCLR_release-(Debian.10.Amd64.Open)Ubuntu.1804.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-10-helix-amd64-20210304164434-56c6673 console.log runclient.py
1913236 runtime PR 73095 net7.0-Linux-Debug-x64-CoreCLR_release-Ubuntu.1804.Amd64.Open console.log runclient.py
1913236 runtime PR 73095 net7.0-Linux-Debug-x64-Mono_release-RedHat.7.Amd64.Open console.log runclient.py
1913236 runtime PR 73095 net7.0-Linux-Debug-x64-Mono_release-(Centos.7.Amd64.Open)Ubuntu.1604.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7-mlnet-helix-20220601183719-dde38af console.log runclient.py
1913236 runtime PR 73095 net7.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.314.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-helix-arm64v8-20210910135810-8a6f4f3 console.log runclient.py
1913236 runtime PR 73095 net7.0-OSX-Debug-x64-Mono_release-OSX.1200.Amd64.Open console.log runclient.py
1913236 runtime PR 73095 net7.0-Linux-Debug-x64-mono_interpreter_release-(Debian.10.Amd64.Open)Ubuntu.1804.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-10-helix-amd64-20210304164434-56c6673 console.log runclient.py
1913058 runtime PR 73037 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py

Build Result Summary

Day Hit Count Week Hit Count Month Hit Count
0 1 10
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Aug 2, 2022
@ghost
Copy link

ghost commented Aug 2, 2022

Tagging subscribers to this area: @dotnet/area-system-security, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

Runfo Creating Tracking Issue (data being generated)

Author: runfoapp[bot]
Assignees: -
Labels:

area-System.Security
Milestone: -

@noahfalk noahfalk added the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Aug 2, 2022
@noahfalk noahfalk changed the title system.security.cryptography.openssl.tests missing libssl CI failure: system.security.cryptography.openssl.tests missing libssl Aug 2, 2022
@bartonjs
Copy link
Member

bartonjs commented Aug 2, 2022

@dotnet/dnceng Looks like some macOS images are bad again.

@andriipatsula
Copy link
Member

Hello @bartonjs , we have a known issue dotnet/arcade#10139 for that. It's related to ARM64 machines, but I believe the problem is the same for amd64 too. I'll link this one to arcade issue.

@andriipatsula andriipatsula mentioned this issue Aug 2, 2022
Closed
@jeffhandley jeffhandley added the tracking-external-issue The issue is caused by external problem (e.g. OS) - nothing we can do to fix it directly label Aug 2, 2022
@jeffhandley jeffhandley added this to the 7.0.0 milestone Aug 2, 2022
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Aug 2, 2022
@runfoapp runfoapp bot mentioned this issue Aug 2, 2022
Closed
@MattGal
Copy link
Member

MattGal commented Aug 2, 2022

As I was working on dotnet/arcade#10139, I will take a look however that's not likely particularly related as that problem was purely with different, erroneous, setup steps on M1 Mac machines.

@MattGal
Copy link
Member

MattGal commented Aug 2, 2022

@bartonjs I fixed up the ARM64 machines yesterday, it would seem we have the same problem on recently imaged 11.00 and 12.00 AMD64 machines as well. dotnet/arcade#10139 is still open and I'll investigate and get those machines fixed today.

I would also note that the OSX failures linked in this issue are all over the place and many are not LibSSL related... that certainly throws a wrench in things.

@wfurt
Copy link
Member

wfurt commented Aug 2, 2022

This is nature of Runfo to bundle unrelated failures together ;(. I trained myself to only look at rolling builds -> there is one hit @MattGal

Console log: 'System.Security.Cryptography.OpenSsl.Tests' from job d48929e0-9acb-474b-aa57-c93c7d1e9f43 workitem 3da839f0-191a-427b-95a1-0dbc7aa55df9 (osx.1200.amd64.open) executed on machine dci-mac-build-103
+ ./RunTests.sh --runtime-path /tmp/helix/working/B16B09B5/p
----- start Sun Jul 31 16:44:59 EDT 2022 =============== To repro directly: =====================================================
pushd .
/tmp/helix/working/B16B09B5/p/dotnet exec --runtimeconfig System.Security.Cryptography.OpenSsl.Tests.runtimeconfig.json --depsfile System.Security.Cryptography.OpenSsl.Tests.deps.json xunit.console.dll System.Security.Cryptography.OpenSsl.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
/private/tmp/helix/working/B16B09B5/w/AB2909A1/e /private/tmp/helix/working/B16B09B5/w/AB2909A1/e
  Discovering: System.Security.Cryptography.OpenSsl.Tests (method display = ClassAndMethod, method display options = None)
No usable version of libssl was found
./RunTests.sh: line 168: 59278 Abort trap: 6           "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Security.Cryptography.OpenSsl.Tests.runtimeconfig.json --depsfile System.Security.Cryptography.OpenSsl.Tests.deps.json

@MattGal
Copy link
Member

MattGal commented Aug 2, 2022

This is nature of Runfo to bundle unrelated failures together ;(. I trained myself to only look at rolling builds -> there is one hit @MattGal

Console log: 'System.Security.Cryptography.OpenSsl.Tests' from job d48929e0-9acb-474b-aa57-c93c7d1e9f43 workitem 3da839f0-191a-427b-95a1-0dbc7aa55df9 (osx.1200.amd64.open) executed on machine dci-mac-build-103
+ ./RunTests.sh --runtime-path /tmp/helix/working/B16B09B5/p
----- start Sun Jul 31 16:44:59 EDT 2022 =============== To repro directly: =====================================================
pushd .
/tmp/helix/working/B16B09B5/p/dotnet exec --runtimeconfig System.Security.Cryptography.OpenSsl.Tests.runtimeconfig.json --depsfile System.Security.Cryptography.OpenSsl.Tests.deps.json xunit.console.dll System.Security.Cryptography.OpenSsl.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
/private/tmp/helix/working/B16B09B5/w/AB2909A1/e /private/tmp/helix/working/B16B09B5/w/AB2909A1/e
  Discovering: System.Security.Cryptography.OpenSsl.Tests (method display = ClassAndMethod, method display options = None)
No usable version of libssl was found
./RunTests.sh: line 168: 59278 Abort trap: 6           "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Security.Cryptography.OpenSsl.Tests.runtimeconfig.json --depsfile System.Security.Cryptography.OpenSsl.Tests.deps.json

Yup, I found the one bad machine about 20 minutes ago and added notes in dotnet/arcade#10139. It should be clean everywhere and I have chatted with the DDFUN folks who set up macs, as well as updated their docs.

@jeffhandley
Copy link
Member

Yup, I found the one bad machine about 20 minutes ago and added notes in dotnet/arcade#10139. It should be clean everywhere and I have chatted with the DDFUN folks who set up macs, as well as updated their docs.

Thanks, @MattGal!

@ghost ghost locked as resolved and limited conversation to collaborators Sep 10, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-System.Security blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' tracking-external-issue The issue is caused by external problem (e.g. OS) - nothing we can do to fix it directly
Projects
None yet
Milestone
7.0.0
Development

No branches or pull requests
6 participants
@jeffhandley @noahfalk @MattGal @bartonjs @wfurt @andriipatsula