Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

System.IO.Compression.Tests.ZipFile_Unix.UnixCreateSetsPermissionsInExternalAttributes fail on OSX #68293

Closed
runfoapp bot opened this issue Apr 20, 2022 · 11 comments · Fixed by #69583
Closed
Assignees
Labels
area-System.IO.Compression disabled-test The test is disabled in source code against the issue
Milestone

Comments

@runfoapp
Copy link

runfoapp bot commented Apr 20, 2022

Starting:    System.IO.Compression.ZipFile.Tests (parallel test collections = on, max threads = 6)
  System.IO.Compression.Tests.ZipFile_Unix.UnixCreateSetsPermissionsInExternalAttributes [FAIL]
    Assert.Equal() Failure
    Expected: 3968
    Actual:   2944
    Stack Trace:
      /_/src/libraries/System.IO.Compression.ZipFile/tests/ZipFile.Unix.cs(43,0): at System.IO.Compression.Tests.ZipFile_Unix.<UnixCreateSetsPermissionsInExternalAttributes>g__EnsureExternalAttributes|0_0(String permissions, ZipArchiveEntry entry)
      /_/src/libraries/System.IO.Compression.ZipFile/tests/ZipFile.Unix.cs(38,0): at System.IO.Compression.Tests.ZipFile_Unix.UnixCreateSetsPermissionsInExternalAttributes()
         at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
      /_/src/coreclr/System.Private.CoreLib/src/System/Reflection/RuntimeMethodInfo.CoreCLR.cs(379,0): at System.Reflection.RuntimeMethodInfo.InvokeNonEmitUnsafe(Object obj, IntPtr* arguments, Span`1 argsForTemporaryMonoSupport, BindingFlags invokeAttr)

Runfo Tracking Issue: System.IO.Compression.Tests.ZipFile_Unix.UnixCreateSetsPermissionsInExternalAttributes fail on OSX

Build Definition Kind Run Name Console Core Dump Test Results Run Client
1780676 runtime PR 69583 net7.0-OSX-Debug-x64-Mono_release-OSX.1200.Amd64.Open console.log runclient.py
1780676 runtime PR 69583 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1780676 runtime PR 69583 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1752830 runtime PR 68876 net6.0-OSX-Debug-x64-Mono_release-OSX.1015.Amd64.Open console.log runclient.py
1752830 runtime PR 68876 net6.0-OSX-Debug-x64-CoreCLR_checked-OSX.1015.Amd64.Open console.log runclient.py
1752830 runtime PR 68876 net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1015.Amd64.Open console.log runclient.py
1752422 runtime Rolling net6.0-OSX-Release-x64-Mono_release-OSX.1015.Amd64.Open console.log runclient.py
1752422 runtime Rolling net6.0-OSX-Release-x64-CoreCLR_release-OSX.1015.Amd64.Open console.log runclient.py
1752422 runtime Rolling net6.0-OSX-Release-x64-CoreCLR_checked-OSX.1015.Amd64.Open console.log runclient.py
1752118 runtime Rolling net6.0-OSX-Release-x64-Mono_release-OSX.1015.Amd64.Open console.log runclient.py
1752118 runtime Rolling net6.0-OSX-Release-x64-CoreCLR_release-OSX.1015.Amd64.Open console.log runclient.py
1752118 runtime Rolling net6.0-OSX-Release-x64-CoreCLR_checked-OSX.1015.Amd64.Open console.log runclient.py
1752112 runtime PR 68352 net6.0-OSX-Debug-x64-Mono_release-OSX.1015.Amd64.Open console.log runclient.py
1752112 runtime PR 68352 net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1015.Amd64.Open console.log runclient.py
1752112 runtime PR 68352 net6.0-OSX-Debug-x64-CoreCLR_checked-OSX.1015.Amd64.Open console.log runclient.py
1751402 runtime PR 67901 net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1015.Amd64.Open console.log runclient.py
1751402 runtime PR 67901 net6.0-OSX-Debug-x64-Mono_release-OSX.1015.Amd64.Open console.log runclient.py
1751402 runtime PR 67901 net6.0-OSX-Debug-x64-CoreCLR_checked-OSX.1015.Amd64.Open console.log runclient.py
1751109 runtime Rolling net6.0-OSX-Release-x64-CoreCLR_release-OSX.1015.Amd64.Open console.log runclient.py
1751109 runtime Rolling net6.0-OSX-Release-x64-Mono_release-OSX.1015.Amd64.Open console.log runclient.py
1751109 runtime Rolling net6.0-OSX-Release-x64-CoreCLR_checked-OSX.1015.Amd64.Open console.log runclient.py
1750872 runtime PR 68425 net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1015.Amd64.Open console.log runclient.py
1750872 runtime PR 68425 net6.0-OSX-Debug-x64-Mono_release-OSX.1015.Amd64.Open console.log runclient.py
1750872 runtime PR 68425 net6.0-OSX-Debug-x64-CoreCLR_checked-OSX.1015.Amd64.Open console.log runclient.py
1750836 runtime Rolling net6.0-OSX-Release-x64-CoreCLR_checked-OSX.1015.Amd64.Open console.log runclient.py
1750836 runtime Rolling net6.0-OSX-Release-x64-CoreCLR_release-OSX.1015.Amd64.Open console.log runclient.py
1750836 runtime Rolling net6.0-OSX-Release-x64-Mono_release-OSX.1015.Amd64.Open console.log runclient.py
1750701 runtime PR 68825 net6.0-OSX-Debug-x64-CoreCLR_checked-OSX.1015.Amd64.Open console.log runclient.py
1750701 runtime PR 68825 net6.0-OSX-Debug-x64-Mono_release-OSX.1015.Amd64.Open console.log runclient.py
1750701 runtime PR 68825 net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1015.Amd64.Open console.log runclient.py
1749891 runtime PR 68825 net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1015.Amd64.Open console.log runclient.py
1749891 runtime PR 68825 net6.0-OSX-Debug-x64-Mono_release-OSX.1015.Amd64.Open console.log runclient.py
1749891 runtime PR 68825 net6.0-OSX-Debug-x64-CoreCLR_checked-OSX.1015.Amd64.Open console.log runclient.py
1744574 runtime Rolling net7.0-OSX-Release-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1744574 runtime Rolling net7.0-OSX-Release-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1744574 runtime Rolling net7.0-OSX-Release-x64-Mono_release-OSX.1200.Amd64.Open console.log runclient.py
1744385 runtime PR 68583 net6.0-OSX-Debug-x64-Mono_release-OSX.1015.Amd64.Open console.log runclient.py
1744385 runtime PR 68583 net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1015.Amd64.Open console.log runclient.py
1744385 runtime PR 68583 net6.0-OSX-Debug-x64-CoreCLR_checked-OSX.1015.Amd64.Open console.log runclient.py
1744068 runtime PR 68695 net7.0-OSX-Debug-x64-Mono_release-OSX.1200.Amd64.Open console.log runclient.py
1744068 runtime PR 68695 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1744068 runtime PR 68695 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1742730 runtime PR 68674 net6.0-OSX-Debug-x64-CoreCLR_checked-OSX.1015.Amd64.Open console.log runclient.py
1742108 runtime PR 68332 net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1015.Amd64.Open console.log runclient.py
1742108 runtime PR 68332 net6.0-OSX-Debug-x64-CoreCLR_checked-OSX.1015.Amd64.Open console.log runclient.py
1742108 runtime PR 68332 net6.0-OSX-Debug-x64-Mono_release-OSX.1015.Amd64.Open console.log runclient.py
1728986 runtime PR 67636 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1728971 runtime PR 68288 net7.0-OSX-Debug-x64-Mono_release-OSX.1200.Amd64.Open console.log runclient.py
1728971 runtime PR 68288 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1728971 runtime PR 68288 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1728898 runtime PR 68241 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1728898 runtime PR 68241 net7.0-OSX-Debug-x64-Mono_release-OSX.1200.Amd64.Open console.log runclient.py
1728898 runtime PR 68241 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1728865 runtime PR 68285 net7.0-OSX-Debug-x64-Mono_release-OSX.1200.Amd64.Open console.log runclient.py
1728865 runtime PR 68285 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1728865 runtime PR 68285 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1728841 runtime PR 68274 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1728841 runtime PR 68274 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1728841 runtime PR 68274 net7.0-OSX-Debug-x64-Mono_release-OSX.1200.Amd64.Open console.log runclient.py
1728819 runtime PR 68283 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1728808 runtime PR 68282 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1728791 runtime PR 68281 net7.0-OSX-Debug-x64-Mono_release-OSX.1200.Amd64.Open console.log runclient.py
1728791 runtime PR 68281 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1728791 runtime PR 68281 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1728743 runtime PR 68177 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1728743 runtime PR 68177 net7.0-OSX-Debug-x64-Mono_release-OSX.1200.Amd64.Open console.log runclient.py
1728743 runtime PR 68177 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1728719 runtime PR 68270 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1728653 runtime PR 67997 net7.0-OSX-Debug-x64-Mono_release-OSX.1200.Amd64.Open console.log runclient.py
1728653 runtime PR 67997 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1728653 runtime PR 67997 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1728630 runtime PR 68275 net7.0-OSX-Debug-x64-Mono_release-OSX.1200.Amd64.Open console.log runclient.py
1728630 runtime PR 68275 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1728630 runtime PR 68275 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1728577 runtime PR 68104 net7.0-OSX-Debug-x64-Mono_release-OSX.1200.Amd64.Open console.log runclient.py
1728577 runtime PR 68104 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1728577 runtime PR 68104 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1728573 runtime PR 68273 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1728573 runtime PR 68273 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1728573 runtime PR 68273 net7.0-OSX-Debug-x64-Mono_release-OSX.1200.Amd64.Open console.log runclient.py
1728533 runtime PR 67701 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1728533 runtime PR 67701 net7.0-OSX-Debug-x64-Mono_release-OSX.1200.Amd64.Open console.log runclient.py
1728533 runtime PR 67701 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1728481 runtime PR 68271 net7.0-OSX-Debug-x64-Mono_release-OSX.1200.Amd64.Open console.log runclient.py
1728481 runtime PR 68271 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1728481 runtime PR 68271 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1728470 runtime PR 60236 net7.0-OSX-Debug-x64-CoreCLR_release-OSX.1200.Amd64.Open console.log runclient.py
1728470 runtime PR 60236 net7.0-OSX-Debug-x64-CoreCLR_checked-OSX.1200.Amd64.Open console.log runclient.py
1728470 runtime PR 60236 net7.0-OSX-Debug-x64-Mono_release-OSX.1200.Amd64.Open console.log runclient.py

Build Result Summary

Day Hit Count Week Hit Count Month Hit Count
1 1 17
@dotnet-issue-labeler dotnet-issue-labeler bot added area-System.IO.Compression untriaged New issue has not been triaged by the area owner labels Apr 20, 2022
@ghost
Copy link

ghost commented Apr 20, 2022

Tagging subscribers to this area: @dotnet/area-system-io-compression
See info in area-owners.md if you want to be subscribed.

Issue Details

Runfo Creating Tracking Issue (data being generated)

Author: runfoapp[bot]
Assignees: -
Labels:

area-System.IO.Compression, untriaged

Milestone: -

@danmoseley
Copy link
Member

The earliest build listed above was 4/20/2022, 11:00:15 AM. I checked in my zip bug fix (not that I can see how it was related) at ~12 PST.

@elinor-fung
Copy link
Member

Expected: 3968 -> 7600, which is S_ISUID | S_ISGID | S_ISVTX
Actual: 2944 -> 5600, which is S_ISUID | S_ISVTX

Lots of hits today. Marking as blocking-clean-ci.

@elinor-fung elinor-fung added the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Apr 20, 2022
@danmoseley
Copy link
Member

@dotnet/dnceng did anything change in the OSX 12 image?

@eerhardt you recall this code -- any idea?

this line may be relevant, but it suggests that we should expect these bits to be cleared?

// S_ISUID, S_ISGID, and S_ISVTX bits. This matches unzip's default behavior.

@MattGal
Copy link
Member

MattGal commented Apr 20, 2022

@dotnet/dnceng did anything change in the OSX 12 image?

@eerhardt you recall this code -- any idea?

this line may be relevant, but it suggests that we should expect these bits to be cleared?

// S_ISUID, S_ISGID, and S_ISVTX bits. This matches unzip's default behavior.

Hi Dan! Can you comment on time? The only interesting thing to change here was moving from 12.0 to 12.3 a while back because the GA 12.0 kept timing out. That wasn't particularly recently though, so your time range would be useful to know.

@MattGal
Copy link
Member

MattGal commented Apr 20, 2022

@danmoseley I confirmed the update (12.0 -> 12.3 latest-at-the-time) was on 3/29, about 3 weeks ago. Beyond this, these machines have done nothing but execute test work items.

@danmoseley
Copy link
Member

@eerhardt thoughts? does this pass on your mac hardware (I assume it's 12.3 or you can update it)

@danmoseley
Copy link
Member

oh, he's out til Monday. I'll disable the test meantime (I should have done this already)

@danmoseley danmoseley added disabled-test The test is disabled in source code against the issue and removed blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' labels Apr 22, 2022
@jozkee jozkee added this to the 7.0.0 milestone May 3, 2022
@jozkee jozkee removed the untriaged New issue has not been triaged by the area owner label May 3, 2022
@carlossanlop
Copy link
Member

carlossanlop commented May 4, 2022

Also affecting the release/6.0 branch. I'm disabling the tests there too: #68867

When this issue gets fixed, the fix needs to get backported to release/6.0.

eerhardt added a commit to eerhardt/runtime that referenced this issue May 19, 2022
…pple OSes

There are scenarios where chmod succeeds, but the OS clears certain bits, like S_ISGID and S_ISUID. When this happens, the ZipFile tests fail because the .zip file doesn't contain the expected ExternalAttributes.

To fix this, read the file mode after calling chmod, and update the expected file mode so the tests still pass.

Fix dotnet#68293
Fix dotnet#60581
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label May 19, 2022
@eerhardt
Copy link
Member

I finally got back to this. I've created #69583 to fix this issue.

The problem, as far as I can tell, is that the macOS devices used in CI must have changed which group they run under, or somehow the user configuration on the machines changed. The tests were failing because they expected the S_ISGID bit to be set on the files, but that bit wasn't be set. Reading https://linux.die.net/man/3/chmod, it says:

If the calling process does not have appropriate privileges, and if the group ID of the file does not match the effective group ID or one of the supplementary group IDs and if the file is a regular file, bit S_ISGID (set-group-ID on execution) in the file's mode shall be cleared upon successful return from chmod().

So, what was happening here is that the test was calling chmod with S_ISGID being set, but the OS was clearing the bit. To fix this I changed the test to read the file mode after calling chmod and using that as the expected file mode. On "normal dev" unix environments (including my local macOS 12.3 machine), the bit isn't being cleared, and the tests pass successfully without modification. But to allow CI to work, the tests will now handle this case.

@carlossanlop
Copy link
Member

carlossanlop commented May 20, 2022

Thanks for the investigation, @eerhardt. Coincidentally, I still have to fix a bug in the tar tests related to chmod to make sure it is working properly, and after the fix I need to make sure they don't hit the same failure.

jozkee pushed a commit that referenced this issue May 20, 2022
…pple OSes (#69583)

* ZipFile UnixCreateSetsPermissionsInExternalAttributes test fails on Apple OSes

There are scenarios where chmod succeeds, but the OS clears certain bits, like S_ISGID and S_ISUID. When this happens, the ZipFile tests fail because the .zip file doesn't contain the expected ExternalAttributes.

To fix this, read the file mode after calling chmod, and update the expected file mode so the tests still pass.

Fix #68293
Fix #60581

* Fix the tests to expect the correct file name
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label May 20, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Jun 19, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-System.IO.Compression disabled-test The test is disabled in source code against the issue
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants