[NativeAOT] Enable CI for macOS x64/arm64 #75421

filipnavara · 2022-09-11T19:08:31Z

No description provided.

ghost · 2022-09-11T19:08:44Z

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

null

Author:	filipnavara
Assignees:	-
Labels:	`area-Infrastructure-libraries`
Milestone:	-

filipnavara · 2022-09-11T19:09:32Z

I assume that someone will need to run appropriate /azp run runtime-extra-platforms-other trigger.

jkotas · 2022-09-11T19:12:50Z

/azp run runtime-extra-platforms

azure-pipelines · 2022-09-11T19:13:01Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2022-09-11T19:13:17Z

No pipelines are associated with this pull request.

filipnavara · 2022-09-11T19:13:18Z

Thanks!

filipnavara · 2022-09-11T19:38:42Z

I realized a lot of tests will fail because #75333 was not merged yet. I also should enable regular NativeAOT osx-arm64 build in the non-extra pipelines. Let's see the results first though...

jkotas · 2022-09-11T20:19:17Z

Looks like this needs fixing:

runtime/eng/testing/tests.singlefile.targets

Line 24 in 25d88b4

    
           <CppCompilerAndLinker Condition="'$(TargetArchitecture)' != '$(BuildArchitecture)' and '$(HostOS)' != 'windows'">clang-9</CppCompilerAndLinker>

filipnavara · 2022-09-11T20:22:16Z

Looks like this needs fixing

Yep. Not quite sure what the correct fix is but I will give it a try.

jkotas · 2022-09-11T20:26:47Z

I think we want to use clang-9 for Linux only (it is a workaround for missing clang alias in our Linux docker images).

filipnavara · 2022-09-11T20:30:43Z

Yep, seems like regular clang (ie. unspecified CppCompilerAndLinker) works for the cross compiled binaries.

jkotas · 2022-09-11T23:56:29Z

The cross-compilation does not work as expected:

2022-09-11T21:40:09.9502410Z   ld: warning: ignoring file /Users/runner/work/1/s/artifacts/bin/coreclr/OSX.arm64.Release/aotsdk/libbootstrapper.a, building for macOS-x86_64 but attempting to link with file built for macOS-arm64

Do we need to pass some kind of target architecture flag to clang?

am11 · 2022-09-12T03:36:56Z

A cross compilation flag (--target) is excluded from TargetOS:OSX, and since --target on OSX is version suffixed (such as, arm64-apple-darwin21.6.0), we prefer the simpler -arch argument.

Try applying this kind of a patch (untested, as I don't have mac with intel chip):

--- a/src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.Unix.targets
+++ b/src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.Unix.targets
@@ -29,11 +29,12 @@ The .NET Foundation licenses this file to you under the MIT license.
       <FullRuntimeName Condition="'$(ServerGarbageCollection)' == 'true'">libRuntime.ServerGC</FullRuntimeName>
 
       <CrossCompileRid />
-      <CrossCompileRid Condition="'$(TargetOS)' != 'OSX' and !$(RuntimeIdentifier.EndsWith('-$(OSHostArch)'))">$(RuntimeIdentifier)</CrossCompileRid>
+      <CrossCompileRid Condition="!$(RuntimeIdentifier.EndsWith('-$(OSHostArch)'))">$(RuntimeIdentifier)</CrossCompileRid>
 
       <CrossCompileArch />
       <CrossCompileArch Condition="$(CrossCompileRid.EndsWith('-x64'))">x86_64</CrossCompileArch>
-      <CrossCompileArch Condition="$(CrossCompileRid.EndsWith('-arm64'))">aarch64</CrossCompileArch>
+      <CrossCompileArch Condition="$(CrossCompileRid.EndsWith('-arm64')) and '$(TargetOS)' != 'OSX'">aarch64</CrossCompileArch>
+      <CrossCompileArch Condition="$(CrossCompileRid.EndsWith('-arm64')) and '$(TargetOS)' == 'OSX'">arm64</CrossCompileArch>
 
       <TargetTriple />
       <TargetTriple Condition="'$(CrossCompileArch)' != ''">$(CrossCompileArch)-linux-gnu</TargetTriple>
@@ -85,7 +86,8 @@ The .NET Foundation licenses this file to you under the MIT license.
     <ItemGroup>
       <LinkerArg Include="@(NativeLibrary)" />
       <LinkerArg Include="--sysroot=$(SysRoot)" Condition="'$(SysRoot)' != ''" />
-      <LinkerArg Include="--target=$(TargetTriple)" Condition="'$(TargetTriple)' != ''" />
+      <LinkerArg Include="--target=$(TargetTriple)" Condition="'$(TargetOS)' != 'OSX' and '$(TargetTriple)' != ''" />
+      <LinkerArg Include="-arch $(CrossCompileArch)" Condition="'$(TargetOS)' == 'OSX' and '$(CrossCompileArch)' != ''" />
       <LinkerArg Include="-g" Condition="$(NativeDebugSymbols) == 'true'" />
       <LinkerArg Include="-Wl,--strip-debug" Condition="$(NativeDebugSymbols) != 'true' and '$(TargetOS)' != 'OSX'" />
       <LinkerArg Include="-Wl,-rpath,'$(IlcRPath)'" />

filipnavara · 2022-09-12T03:41:57Z

untested, as I don't have mac with intel chip

I will test it today. You can run the whole build under arch -arch x86-64 ./build.sh ... though. That should allow testing the scenario even on M1/M2 Macs.

am11 · 2022-09-12T04:38:07Z

It took these changes to succeed the cross build: am11@542104d.

You can run the whole build under arch -arch x86-64 ./build.sh ... though

Good to know. It was -arch x86_64 though.

filipnavara · 2022-09-12T08:24:24Z

It got past the point of previous failure. We will need to re-trigger the extra pipeline.

jkotas · 2022-09-12T14:10:39Z

/azp run runtime-extra-platforms

azure-pipelines · 2022-09-12T14:11:22Z

Azure Pipelines successfully started running 1 pipeline(s).

filipnavara · 2022-09-12T17:46:51Z

macOS x64 is still crash fest unfortunately. I will check the dumps but if I don't find anything obvious I will split that off into separate PR.

macOS ARM64 all seem like small things that should be easy to fix. I plan to do it later this week.

filipnavara · 2022-09-13T08:29:12Z

macOS arm64:

System.Memory tests crash could not be reproduced locally on my machine [even with the exact Helix payload]. It seems we don't do dumps on Helix for NativeAOT so there's no information to diagnose at the moment.
System.IO.MemoryMappedFiles tests only failed for cross-compilation. I pushed a fix.
OSVersion_ValidVersion_OSX test relies on how CoreCLR fills the RID information. NativeAOT hard-codes it during compilation process. I disabled the test since it was supposed to test Environment.OSVersion (which works) and not RuntimeInformation.RuntimeIdentifier.

macOS x64:
I can reproduce the crashes locally but it's basically undebuggable. When running under lldb I run into loop in SuspendAllThreads that seems to be never ending. It keeps trying to hijack threads through PalHijack which sends signal to the target thread. Since lldb is slower with delivering the signals it keeps calling PalHijack so frequently that ActivationHandler doesn't seem to have time to run. I am not sure if it's something specific to lldb or Rosetta but it seems bit fishy to me. cc @janvorli

UPD: The macOS x64 may have problem stack walking but it could easily be something in the context capture too... The Debug build has meaningful asserts at least.

UPD 2: There's definitely garbage compact unwinding information. In some cases it encodes stack size as reference into the code but it points to instruction that's not subl $xxx,%esp (functionStart is pointing into middle of the code) and computes grabage. This garbage is then used as relative offset to stack and causes the SIGSEGVs.

UPD 3: The incorrect functionStart is actually bug in my code, it just was not used on arm64 so it went unnoticed. Should be a one-line fix.

UPD 4: With the fixed compact unwinding I am back to failing thread suspension/hijacking. In debug builds I also get asserts from Rosetta all the time when running under lldb (or the hang mentioned above). When running without debugger it crashes with:

Thread 13 Crashed:
0   ???                           	    0x7ff896c7a9a8 ???
1   libsystem_kernel.dylib        	    0x7ff80790f30e __pthread_kill + 10
2   libsystem_pthread.dylib       	    0x7ff807946f7b pthread_kill + 263
3   libsystem_c.dylib             	    0x7ff807890ca5 abort + 123
4   System.Collections.Tests      	       0x104d754ec Assert(char const*, char const*, unsigned int, char const*) + 28 (rhassert.cpp:24)
5   System.Collections.Tests      	       0x104d8e264 Thread::HijackReturnAddressWorker(StackFrameIterator*, void (*)()) + 276 (thread.cpp:789)
6   System.Collections.Tests      	       0x104d8ddee Thread::HijackReturnAddress(UNIX_CONTEXT*, void (*)()) + 174 (thread.cpp:764)
7   System.Collections.Tests      	       0x104d8dc85 Thread::HijackCallback(UNIX_CONTEXT*, void*) + 485 (thread.cpp:689)
8   System.Collections.Tests      	       0x104e0fbb6 ActivationHandler(int, __siginfo*, void*) + 102 (PalRedhawkUnix.cpp:1021)
9   libsystem_platform.dylib      	    0x7ff807971c1d _sigtramp + 29
10  ???                           	       0x11c00e680 ???

jkotas · 2022-09-13T12:22:22Z

With the fixed compact unwinding I am back to failing thread suspension/hijacking. In debug builds I also get asserts from Rosetta all the time when running under lldb (or the hang mentioned above).

@VSadov Could you please take a look?

src/libraries/System.Runtime.Extensions/tests/System/EnvironmentTests.cs

filipnavara · 2022-09-13T13:10:45Z

With the fixed compact unwinding I am back to failing thread suspension/hijacking. In debug builds I also get asserts from Rosetta all the time when running under lldb (or the hang mentioned above).

@VSadov Could you please take a look?

My theory is that the underlying reason why the hijacking fails is that the stack unwinding doesn't quite work correctly for this particular use case. The compact unwinding may not be accurate when unwinding a frame which is currently in the prolog/epilog. This likely happens significantly more often in the Rosetta emulation layer where I seem to hit the asserts with the signal being delivered exactly at the first instruction of a method. In fact, not even lldb can unwind these frames in the bt command.

On ARM64 this doesn't happen, or at least not that often, because most of the managed methods get compiled with prologs/epilogs that cannot be represented as compact unwinding and thus it falls back to DWARF unwinding information in the end (compact unwinding is still needed to make the linker happy and insert pointers to the DWARF info). DWARF has enough information to unwind even prologs/epilogs correctly.

filipnavara · 2022-09-13T14:44:54Z

I added some rudimentary handling of the function prologs in UnixNativeCodeManager::GetReturnAddressHijackInfo. It seems to be good enough to let me finish the tests that were previously failing on osx-x64. I didn't do extensive testing but I guess it would interesting to have another go at /azp run runtime-extra-platforms.

The test tries to validate Environment.OSVersion.Version which works correctly. RuntimeInformation.RuntimeIdentifier, however, doesn't use versioned RIDs when NativeAOT is used.

…n in triple to get compact unwinding tables generated

filipnavara · 2022-09-30T08:24:09Z

Rebased on top of main and manually bumped ObjWriter packages (maestro merges are currently blocked by build failures). Should be ready for another /azp run runtime-extra-platforms run.

jkotas · 2022-09-30T13:08:00Z

/azp run runtime-extra-platforms

azure-pipelines · 2022-09-30T13:08:19Z

Azure Pipelines successfully started running 1 pipeline(s).

filipnavara · 2022-09-30T15:41:05Z

(+) No more random crashes.
(-) ~~Few more interop tests need a tweak~~
(-) The ManyConcurrentAddsTakes_ForceContentionWithToArray failure on osx-arm64 may be real problem

The AzDO link doesn't seem to work anymore. Here are the details:

 "HelixJobId": "9096e23d-ea87-461c-b90d-4f1a203edfcb", "HelixWorkItemName": "System.Collections.Concurrent.Tests" }


Error message
Assert.DoesNotContain() Failure
Found:    0
In value: Int32[] [0]


Stack trace
   at System.Collections.Concurrent.Tests.ProducerConsumerCollectionTests.ManyConcurrentAddsTakes_ForceContentionWithToArray(Double seconds) + 0x110
   at System.Collections.Concurrent!<BaseAddress>+0x9fe65c
   at System.Reflection.DynamicInvokeInfo.Invoke(Object, IntPtr, Object[], BinderBundle, Boolean) + 0x148

jkotas · 2022-09-30T15:58:30Z

/azp run runtime-extra-platforms

azure-pipelines · 2022-09-30T15:58:43Z

Azure Pipelines successfully started running 1 pipeline(s).

filipnavara · 2022-09-30T20:00:45Z

The test failures seem unrelated (tvOS succeeded but failed to pass the XML results due to TCP port in use; Linux Bionic failed to find a device) 🎉

jkotas

Looks great. Thank you!

...time.InteropServices.UnitTests/System/Runtime/InteropServices/ObjectiveC/MessageSendTests.cs

…ime.InteropServices.UnitTests/System/Runtime/InteropServices/ObjectiveC/MessageSendTests.cs

filipnavara · 2022-09-30T21:41:17Z

Thanks for all the help on this one!

VSadov · 2022-09-30T23:12:08Z

@filipnavara Thank you!!

dotnet-issue-labeler bot added the area-Infrastructure-libraries label Sep 11, 2022

ghost added the community-contribution Indicates that the PR has been added by a community member label Sep 11, 2022

This comment was marked as outdated.

Sign in to view

filipnavara force-pushed the patch-7 branch from 420329b to 2913429 Compare September 11, 2022 20:38

jkotas reviewed Sep 13, 2022

View reviewed changes

src/libraries/System.Runtime.Extensions/tests/System/EnvironmentTests.cs Outdated Show resolved Hide resolved

filipnavara force-pushed the patch-7 branch from fd4eef3 to 656abda Compare September 13, 2022 12:32

filipnavara and others added 7 commits September 30, 2022 10:07

[NativeAOT] Enable CI for macOS x64/arm64

721a247

Update CppCompilerAndLinker condition

d5c4cb9

Enable NativeAOT osx-arm64 build in regular runtime pipeline

5f2c51b

Enable cross-compilation on macOS

c49d7bb

Fix HAVE_SHM_OPEN_THAT_WORKS_WELL_ENOUGH_WITH_MMAP when cross-compiling

c08a6b4

Update OSVersion_ValidVersion_OSX test on NativeAOT

6a1c982

The test tries to validate Environment.OSVersion.Version which works correctly. RuntimeInformation.RuntimeIdentifier, however, doesn't use versioned RIDs when NativeAOT is used.

Add hack to stop linker from corrupting DWARF CFI, bump darwin versio…

07b2b3a

…n in triple to get compact unwinding tables generated

filipnavara force-pushed the patch-7 branch from 22a8804 to 6a6cb0f Compare September 30, 2022 08:21

Bump ObjWriter packages

d4a521a

filipnavara force-pushed the patch-7 branch from 6a6cb0f to d4a521a Compare September 30, 2022 09:19

Disable MessageSendTests on NativeAOT

fc84074

filipnavara marked this pull request as ready for review September 30, 2022 15:48

filipnavara requested a review from MichalStrehovsky as a code owner September 30, 2022 15:48

jkotas approved these changes Sep 30, 2022

View reviewed changes

jkotas reviewed Sep 30, 2022

View reviewed changes

...time.InteropServices.UnitTests/System/Runtime/InteropServices/ObjectiveC/MessageSendTests.cs Outdated Show resolved Hide resolved

Update src/libraries/System.Runtime.InteropServices/tests/System.Runt…

f652a3c

…ime.InteropServices.UnitTests/System/Runtime/InteropServices/ObjectiveC/MessageSendTests.cs

jkotas merged commit c8f9f29 into dotnet:main Sep 30, 2022

filipnavara mentioned this pull request Sep 30, 2022

Native AOT in .NET 8 #69739

Closed

6 tasks

filipnavara deleted the patch-7 branch October 1, 2022 08:36

jkotas mentioned this pull request Oct 2, 2022

ManyConcurrentAddsTakes_ForceContentionWithToArray fails intermittently on osx-arm64 #76501

Closed

ghost locked as resolved and limited conversation to collaborators Oct 31, 2022

[NativeAOT] Enable CI for macOS x64/arm64 #75421

[NativeAOT] Enable CI for macOS x64/arm64 #75421

Conversation

filipnavara commented Sep 11, 2022

ghost commented Sep 11, 2022

filipnavara commented Sep 11, 2022

This comment was marked as outdated.

This comment was marked as outdated.

jkotas commented Sep 11, 2022

azure-pipelines bot commented Sep 11, 2022

azure-pipelines bot commented Sep 11, 2022

filipnavara commented Sep 11, 2022

filipnavara commented Sep 11, 2022 • edited Loading

jkotas commented Sep 11, 2022

filipnavara commented Sep 11, 2022

jkotas commented Sep 11, 2022

filipnavara commented Sep 11, 2022

jkotas commented Sep 11, 2022

am11 commented Sep 12, 2022 • edited Loading

filipnavara commented Sep 12, 2022

am11 commented Sep 12, 2022

filipnavara commented Sep 12, 2022

jkotas commented Sep 12, 2022

azure-pipelines bot commented Sep 12, 2022

filipnavara commented Sep 12, 2022

filipnavara commented Sep 13, 2022 • edited Loading

jkotas commented Sep 13, 2022

filipnavara commented Sep 13, 2022 • edited Loading

filipnavara commented Sep 13, 2022

filipnavara commented Sep 30, 2022

jkotas commented Sep 30, 2022

azure-pipelines bot commented Sep 30, 2022

filipnavara commented Sep 30, 2022 • edited Loading

jkotas commented Sep 30, 2022

azure-pipelines bot commented Sep 30, 2022

filipnavara commented Sep 30, 2022

jkotas left a comment

Choose a reason for hiding this comment

filipnavara commented Sep 30, 2022

VSadov commented Sep 30, 2022

filipnavara commented Sep 11, 2022 •

edited

Loading

am11 commented Sep 12, 2022 •

edited

Loading

filipnavara commented Sep 13, 2022 •

edited

Loading

filipnavara commented Sep 13, 2022 •

edited

Loading

filipnavara commented Sep 30, 2022 •

edited

Loading