Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump LLVM to 19.x branch #103585

Merged
merged 20 commits into from
Jul 22, 2024
Merged

Bump LLVM to 19.x branch #103585

merged 20 commits into from
Jul 22, 2024

Conversation

radekdoulik
Copy link
Member

No description provided.

@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Jun 17, 2024
@radekdoulik
Copy link
Member Author

@lambdageek, would you have time to look at porting mono to llvm 19.x?

It uses our packages from general testing channel built with unreleased llvm 19.x, same commit as emscripten 3.1.56 uses plus our changes re-based on top of it, https://github.com/dotnet/llvm-project/tree/dotnet/main-19.x

/cc @lewing

@lambdageek
Copy link
Member

@radekdoulik Not sure. /cc @steveisok

Looks like the build failure is due to
llvm/llvm-project@f7ca013

which removed the C bindings to the legacy pass manager. I don't know a ton about this part of Mono.

@lambdageek lambdageek added area-Codegen-LLVM-mono and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Jun 17, 2024
Copy link
Contributor

Tagging subscribers to this area: @lambdageek, @steveisok
See info in area-owners.md if you want to be subscribed.

@lambdageek
Copy link
Member

@radekdoulik can you update this:

https://github.com/dotnet/llvm-project/blob/dotnet/main-19.x/llvm/CMakeLists.txt#L22

I'm trying to do a local build of this PR with the new pass manager commented out, so that I can check what else is broken

@lambdageek
Copy link
Member

@radekdoulik here's some commits to at least get the thing building
https://github.com/radekdoulik/runtime/compare/pr-bump-llvm-19...lambdageek:runtime:radek-pr-bump-llvm-19?expand=1

this isn't a good long term solution, but it will at least get us to the point where we can see what else needs to be fixed up

@lambdageek
Copy link
Member

oh all this LLVM pass manager stuff is for the LLVM JIT - we should just drop support for it.

@lambdageek
Copy link
Member

Made a half-hearted attempt at supporting the new passbuilder for the LLVM JIT. I'm pretty sure it doesn't work (I don't know how to get a LLVMTargetMachineRef without seemingly a ton of work), but it at least builds. If it did work, I'm pretty sure it would run optimizations on the whole JIT module every single time (instead of just a single function). I think it's not worth it.

https://github.com/lambdageek/runtime/commits/radek-pr-bump-llvm-19-plus-passbuilder/


Even if we don't do the new passbuilder, we do need to update this stuff for LLVM-19. The runtime linked without updates for me if I don't touch the new passbuilder, but I'm sure something else would break.

# llvm-config --libs analysis core bitwriter mcjit orcjit
set(MONO_llvm_core_libs "LLVMOrcJIT" "LLVMPasses" "LLVMCoroutines" "LLVMipo" "LLVMInstrumentation" "LLVMVectorize" "LLVMScalarOpts" "LLVMLinker" "LLVMIRReader" "LLVMAsmParser" "LLVMInstCombine" "LLVMFrontendOpenMP" "LLVMAggressiveInstCombine" "LLVMTransformUtils" "LLVMJITLink" "LLVMMCJIT" "LLVMExecutionEngine" "LLVMTarget" "LLVMRuntimeDyld" "LLVMBitWriter" "LLVMAnalysis" "LLVMProfileData" "LLVMObject" "LLVMTextAPI" "LLVMMCParser" "LLVMMC" "LLVMDebugInfoCodeView" "LLVMBitReader" "LLVMCore" "LLVMRemarks" "LLVMBitstreamReader" "LLVMBinaryFormat" "LLVMSupport" "LLVMDemangle")
if(${llvm_api_version} LESS 1600)
set(MONO_llvm_core_libs ${MONO_llvm_core_libs} "LLVMObjCARCOpts" "LLVMMCDisassembler" "LLVMOrcTargetProcess" "LLVMOrcShared" "LLVMDebugInfoDWARF")
else()
set(MONO_llvm_core_libs ${MONO_llvm_core_libs} "LLVMIRPrinter" "LLVMCodeGen" "LLVMObjCARCOpts" "LLVMMCDisassembler" "LLVMWindowsDriver" "LLVMOption" "LLVMOrcTargetProcess" "LLVMOrcShared" "LLVMSymbolize" "LLVMDebugInfoPDB" "LLVMDebugInfoMSF" "LLVMDebugInfoDWARF" "LLVMTargetParser")
endif()
# llvm-config --libs x86codegen
set(MONO_llvm_extra_libs_x86codegen "LLVMX86CodeGen" "LLVMCFGuard" "LLVMGlobalISel" "LLVMX86Desc" "LLVMX86Info" "LLVMMCDisassembler" "LLVMSelectionDAG" "LLVMAsmPrinter" "LLVMDebugInfoDWARF" "LLVMCodeGen" "LLVMTarget" "LLVMScalarOpts" "LLVMInstCombine" "LLVMAggressiveInstCombine" "LLVMTransformUtils" "LLVMBitWriter" "LLVMAnalysis" "LLVMProfileData" "LLVMObject" "LLVMTextAPI" "LLVMMCParser" "LLVMMC" "LLVMDebugInfoCodeView" "LLVMDebugInfoMSF" "LLVMBitReader" "LLVMCore" "LLVMRemarks" "LLVMBitstreamReader" "LLVMBinaryFormat" "LLVMSupport" "LLVMDemangle")
if(${llvm_api_version} GREATER_EQUAL 1600)
set(MONO_llvm_extra_libs_x86codegen ${MONO_llvm_extra_libs_x86codegen} "LLVMInstrumentation" "LLVMObjCARCOpts" "LLVMSymbolize" "LLVMDebugInfoPDB" "LLVMIRReader" "LLVMAsmParser" "LLVMTargetParser")
endif()
# llvm-config --libs armcodegen
set(MONO_llvm_extra_libs_armcodegen "LLVMARMCodeGen" "LLVMCFGuard" "LLVMGlobalISel" "LLVMSelectionDAG" "LLVMAsmPrinter" "LLVMDebugInfoDWARF" "LLVMCodeGen" "LLVMTarget" "LLVMScalarOpts" "LLVMInstCombine" "LLVMAggressiveInstCombine" "LLVMTransformUtils" "LLVMBitWriter" "LLVMAnalysis" "LLVMProfileData" "LLVMObject" "LLVMTextAPI" "LLVMMCParser" "LLVMBitReader" "LLVMCore" "LLVMRemarks" "LLVMBitstreamReader" "LLVMARMDesc" "LLVMMCDisassembler" "LLVMMC" "LLVMDebugInfoCodeView" "LLVMDebugInfoMSF" "LLVMBinaryFormat" "LLVMARMUtils" "LLVMARMInfo" "LLVMSupport" "LLVMDemangle")
if(${llvm_api_version} GREATER_EQUAL 1600)
set(MONO_llvm_extra_libs_armcodegen ${MONO_llvm_extra_libs_armcodegen} "LLVMipo" "LLVMInstrumentation" "LLVMVectorize" "LLVMLinker" "LLVMFrontendOpenMP" "LLVMObjCARCOpts" "LLVMSymbolize" "LLVMDebugInfoPDB" "LLVMIRReader" "LLVMAsmParser" "LLVMTargetParser")
endif()
# llvm-config --libs aarch64codegen
set(MONO_llvm_extra_libs_aarch64codegen "LLVMAArch64CodeGen" "LLVMCFGuard" "LLVMGlobalISel" "LLVMSelectionDAG" "LLVMAsmPrinter" "LLVMDebugInfoDWARF" "LLVMCodeGen" "LLVMTarget" "LLVMScalarOpts" "LLVMInstCombine" "LLVMAggressiveInstCombine" "LLVMTransformUtils" "LLVMBitWriter" "LLVMAnalysis" "LLVMProfileData" "LLVMObject" "LLVMTextAPI" "LLVMMCParser" "LLVMBitReader" "LLVMCore" "LLVMRemarks" "LLVMBitstreamReader" "LLVMAArch64Desc" "LLVMMC" "LLVMDebugInfoCodeView" "LLVMDebugInfoMSF" "LLVMBinaryFormat" "LLVMAArch64Utils" "LLVMAArch64Info" "LLVMSupport" "LLVMDemangle")
if(${llvm_api_version} GREATER_EQUAL 1600)
set(MONO_llvm_extra_libs_aarch64codegen ${MONO_llvm_extra_libs_aarch64codegen} "LLVMObjCARCOpts" "LLVMSymbolize" "LLVMDebugInfoPDB" "LLVMIRReader" "LLVMAsmParser" "LLVMTargetParser")
endif()

@lambdageek
Copy link
Member

@radekdoulik here's some commits to at least get the thing building https://github.com/radekdoulik/runtime/compare/pr-bump-llvm-19...lambdageek:runtime:radek-pr-bump-llvm-19?expand=1

this isn't a good long term solution, but it will at least get us to the point where we can see what else needs to be fixed up

doing:

$ make -c src/mono/sample/HelloWorld clean publish run AOT=true USE_LLVM=true

crashes in llcwith:

//Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : LLVM ERROR: Unsupported calling convention.
/Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
/Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : Stack dump:
/Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : 0.	Program arguments: /Users/alklig/work/dotnet-runtime/runtime-bugfix/artifacts/bin/mono/osx.arm64.Debug/cross/osx-arm64/llc -enable-implicit-null-checks -disable-fault-maps -march=aarch64 -asm-verbose=false -disable-gnu-eh-frame -enable-mono-eh-frame -mono-eh-frame-symbol=_mono_aot_HelloWorld_eh_frame -disable-tail-calls -relocation-model=pic -filetype=obj -o /Users/alklig/work/dotnet-runtime/runtime-bugfix/artifacts/obj/mono/HelloWorld/osx.arm64.Debug/osx-arm64/HelloWorld.dll.tmp/temp-llvm.o /Users/alklig/work/dotnet-runtime/runtime-bugfix/artifacts/obj/mono/HelloWorld/osx.arm64.Debug/osx-arm64/HelloWorld.dll.tmp/temp.opt.bc
/Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : 1.	Running pass 'Function Pass Manager' on module '/Users/alklig/work/dotnet-runtime/runtime-bugfix/artifacts/obj/mono/HelloWorld/osx.arm64.Debug/osx-arm64/HelloWorld.dll.tmp/temp.opt.bc'.
/Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : 2.	Running pass 'AArch64 Instruction Selection' on function '@HelloWorld_Program_Main_string__'
/Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
/Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : 0  llc                      0x000000010214f950 llvm::SmallVectorBase<unsigned long long>::set_size(unsigned long) + 388512
/Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : 1  llc                      0x000000010214dc40 llvm::SmallVectorBase<unsigned long long>::set_size(unsigned long) + 381072
/Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : 2  llc                      0x000000010214ffd4 llvm::SmallVectorBase<unsigned long long>::set_size(unsigned long) + 390180
/Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : 3  libsystem_platform.dylib 0x00000001990b7584 _sigtramp + 56
/Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : 4  libsystem_pthread.dylib  0x0000000199086c20 pthread_kill + 288
/Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : 5  libsystem_c.dylib        0x0000000198f93a30 abort + 180
/Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : 6  llc                      0x00000001020c039c llvm::cl::opt<bool, false, llvm::cl::parser<bool>>::setCallback(std::__1::function<void (bool const&)>) + 116292
/Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : 7  llc                      0x00000001020c01c8 llvm::cl::opt<bool, false, llvm::cl::parser<bool>>::setCallback(std::__1::function<void (bool const&)>) + 115824
/Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : 8  llc                      0x0000000100e6e608
/Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : 9  llc                      0x0000000100e6ecf8
/Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : 10 llc                      0x0000000101fbe5a4 llvm::IntervalMap<long long, std::__1::monostate, 8u, llvm::IntervalMapHalfOpenInfo<long long>>::deleteNode(llvm::IntervalMapImpl::NodeRef, unsigned int) + 1006296
/Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : 11 llc                      0x000000010201c558 bool llvm::ISD::matchUnaryPredicateImpl<llvm::ConstantFPSDNode>(llvm::SDValue, std::__1::function<bool (llvm::ConstantFPSDNode*)>, bool) + 339848
/Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : 12 llc                      0x000000010201b4a0 bool llvm::ISD::matchUnaryPredicateImpl<llvm::ConstantFPSDNode>(llvm::SDValue, std::__1::function<bool (llvm::ConstantFPSDNode*)>, bool) + 335568
/Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : 13 llc                      0x000000010164a278 llvm::Pass* llvm::callDefaultCtor<llvm::MachineDominatorTree, true>() + 135620
/Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : 14 llc                      0x0000000101a39870 llvm::Attribute llvm::CallBase::getFnAttrOnCalledFunction<llvm::StringRef>(llvm::StringRef) const + 171084
/Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : 15 llc                      0x0000000101a3f224 llvm::Attribute llvm::CallBase::getFnAttrOnCalledFunction<llvm::StringRef>(llvm::StringRef) const + 194048
/Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : 16 llc                      0x0000000101a39dec llvm::Attribute llvm::CallBase::getFnAttrOnCalledFunction<llvm::StringRef>(llvm::StringRef) const + 172488
/Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : 17 llc                      0x0000000100a5f864
/Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : 18 llc                      0x0000000100a5d56c
/Users/alklig/work/dotnet-runtime/runtime-bugfix/src/mono/sample/HelloWorld/HelloWorld.csproj(28,5): error : 19 dyld                     0x0000000198cfe0e0 start + 2360

Do we have the mono calling convention patches on the 19.x branch?

radekdoulik and others added 3 commits June 27, 2024 17:00
They dont' have a C API for it anymore.

We should use the new PassBuilder API instead.  But temporarily just
disable optimizations entirely
This reverts commit efca5a2.
Remove coherency between llvm and emscripten to be able to update llvm 19.x packages with darc
@lewing
Copy link
Member

lewing commented Jul 22, 2024

/azp run runtime-ioslike

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

NuGet.config Outdated Show resolved Hide resolved
@lewing
Copy link
Member

lewing commented Jul 22, 2024

Previous build was green enough, we plan to merge.

@lewing
Copy link
Member

lewing commented Jul 22, 2024

/ba-g fast merging to unblock flow from other repos

@lewing lewing merged commit fc6ce2c into dotnet:main Jul 22, 2024
111 of 160 checks passed
@@ -17,6 +17,9 @@
<!-- Active issue: https://github.com/dotnet/runtime/issues/97809 -->
<ShouldILStrip>false</ShouldILStrip>
</PropertyGroup>
<PropertyGroup Condition="'$(TargetOS)' == 'browser'">
<XunitShowProgress>true</XunitShowProgress>
</PropertyGroup>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be reverted

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to keep it in, just in case the issue reappear, to have more information about which test is causing it.

Copy link
Member

@akoeplinger akoeplinger Jul 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We recently removed the progress printing since it was thought to cause timeouts: #103621

I'm fine if we keep it for a couple days but we should ultimately remove it

</ExcludeList>
<ExcludeList Include = "$(XunitTestBinBase)/JIT/Methodical/Methodical_ro/**">
<Issue>Crashes during LLVM AOT compilation.</Issue>
</ExcludeList>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need to file an issue for these

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you remember where did it fail?

Copy link
Member

@matouskozak matouskozak Jul 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interestingly, it's failing on fullAOT-mini lane as well #105395. It's possible that it's related to another change in the range since there were a few of [mono] changes during the period.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apparently this is an incorrect way to disable tests with aot. I used to get failures with these tests on llvm fix PR #105867. Testing locally there were failures for missing aot image because the tests where not being aot compiled but we still tried to run them. Together with the other llvm fixes, enabling these back fixed the failures.

@matouskozak
Copy link
Member

matouskozak commented Aug 6, 2024

Is it possible we are seeing a 0.2MB size increase on iOS due to the LLVM 19 bump ?
image

This is the range fdef8db...d46fdc1 and I don't see anything else suspicious.

@akoeplinger
Copy link
Member

Definitely possible yeah

@matouskozak
Copy link
Member

Definitely possible yeah

Do we use -O3 by default for iOS targets? Maybe we could try using -O2 or (-Oz/s) to keep the assembly size smaller.

@akoeplinger
Copy link
Member

changing the optimization flags can have other unintended consequences. tbh I wouldn't worry about it for a 200k diff on iOS.

@steveisok
Copy link
Member

changing the optimization flags can have other unintended consequences. tbh I wouldn't worry about it for a 200k diff on iOS.

I agree - I think we can live with it

@matouskozak
Copy link
Member

changing the optimization flags can have other unintended consequences. tbh I wouldn't worry about it for a 200k diff on iOS.

I agree - I think we can live with it

I wasn't only thinking about this regression but the overall size increase we were experiencing during .NET 9 cycle (#105701). However, it's just an idea, improvements to trimming and such are likely a better approach.

@github-actions github-actions bot locked and limited conversation to collaborators Sep 11, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants