Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to build on MacOS #145

Open
marcpabst opened this issue Jul 28, 2024 · 38 comments
Open

Unable to build on MacOS #145

marcpabst opened this issue Jul 28, 2024 · 38 comments

Comments

@marcpabst
Copy link

marcpabst commented Jul 28, 2024

Hi, enzyme looks super cool, and now that it’s on its way into nightly, I’d like to play around with it a bit. I’m following the instructions in the README, but the build always errors with the following output:

[92/93] Install the project...
-- Install configuration: "Release"
-- Installing: /Users/X/rust/build/build/aarch64-apple-darwin/enzyme/lib/cmake/Enzyme/EnzymeConfig.cmake
-- Installing: /Users/X/rust/build/build/aarch64-apple-darwin/enzyme/lib/cmake/Enzyme/EnzymeConfigVersion.cmake
-- Installing: /Users/X/rust/build/build/aarch64-apple-darwin/enzyme/lib/cmake/Enzyme/EnzymeTargets.cmake
-- Installing: /Users/X/rust/build/build/aarch64-apple-darwin/enzyme/lib/cmake/Enzyme/EnzymeTargets-release.cmake
-- Installing: /Users/X/rust/build/build/aarch64-apple-darwin/enzyme/lib/LLVMEnzyme-17.dylib
-- Installing: /Users/X/rust/build/build/aarch64-apple-darwin/enzyme/lib/ClangEnzyme-17.dylib
-- Installing: /Users/X/rust/build/build/aarch64-apple-darwin/enzyme/lib/LLDEnzyme-17.dylib
cargo:root=/Users/X/rust/build/build/aarch64-apple-darwin/enzyme
	finished in 61.092 seconds
thread 'main' panicked at src/lib.rs:1605:24:
src.symlink_metadata() failed with No such file or directory (os error 2)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Build completed unsuccessfully in 0:18:44

Any idea how to address this? I'm on an a Macbook Pro M2

Edit: Looks like I'm not the only one with this problem: #27 (comment)

@ZuseZ4
Copy link
Member

ZuseZ4 commented Jul 30, 2024

Hi, yes indeed I have been ignoring MacOS support a bit since I had no good way of debugging it. I am happy to take an hour one of these days to sit down for some co-programming and adjust the paths and file endings to whatever Apple prefers, but other than that I have no good way of making progress here beside of waiting for PRs fixing this. Would you be interested to work on it?

@marcpabst
Copy link
Author

Hey, I’m generally very happy to help with debugging path problems, as long as no understanding of compiler internals or LLVM stuff is needed!

@ZuseZ4
Copy link
Member

ZuseZ4 commented Jul 30, 2024

Yep no worries that's not needed, the bugs are likely that I either hardcoded some paths or file endings that don't exist on linux. So if you are fine with screen-sharing, we can just compile it on your PC with some debug flags and fix all the bugs along the way. Do either Thursday or Friday after 8am PDT work for you?

@marcpabst
Copy link
Author

Great, Thursday 8am PDT / 3pm UTC works! I can send you a Teams link if that's ok!

@marcpabst
Copy link
Author

Here is the full stack trace:

thread 'main' panicked at src/lib.rs:1605:24:
src.symlink_metadata() failed with No such file or directory (os error 2)
stack backtrace:
   0: rust_begin_unwind
             at /rustc/0e09125c6c3c2fd70d7de961bcf0e51575235fad/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/0e09125c6c3c2fd70d7de961bcf0e51575235fad/library/core/src/panicking.rs:72:14
   2: bootstrap::Build::copy_internal
             at ./src/bootstrap/src/lib.rs:1605:24
   3: bootstrap::Build::copy
             at ./src/bootstrap/src/lib.rs:1593:9
   4: <bootstrap::core::build_steps::compile::Assemble as bootstrap::core::builder::Step>::run
             at ./src/bootstrap/src/core/build_steps/compile.rs:1657:13
   5: bootstrap::core::builder::Builder::ensure
             at ./src/bootstrap/src/core/builder.rs:2192:23
   6: bootstrap::core::builder::Builder::compiler
             at ./src/bootstrap/src/core/builder.rs:989:9
   7: <bootstrap::core::build_steps::compile::Std as bootstrap::core::builder::Step>::make_run
             at ./src/bootstrap/src/core/build_steps/compile.rs:107:23
   8: bootstrap::core::builder::StepDescription::maybe_run
             at ./src/bootstrap/src/core/builder.rs:328:13
   9: bootstrap::core::builder::StepDescription::run
             at ./src/bootstrap/src/core/builder.rs:398:17
  10: bootstrap::core::builder::Builder::run_step_descriptions
             at ./src/bootstrap/src/core/builder.rs:981:9
  11: bootstrap::core::builder::Builder::execute_cli
             at ./src/bootstrap/src/core/builder.rs:962:9
  12: bootstrap::Build::build
             at ./src/bootstrap/src/lib.rs:672:13
  13: bootstrap::main
             at ./src/bootstrap/src/bin/main.rs:85:5
  14: core::ops::function::FnOnce::call_once
             at /rustc/0e09125c6c3c2fd70d7de961bcf0e51575235fad/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Build completed unsuccessfully in 0:14:44

@ZuseZ4
Copy link
Member

ZuseZ4 commented Aug 1, 2024

rust/src/bootstrap/src/core/build_steps/compile.rs
line 1652

@marcpabst
Copy link
Author

Changing LINK_FLAGS "-Wl,-flat_namespace -Wl,-undefined -Wl,suppress") to LINK_FLAGS "-dynamiclib -Wl,-undefined,dynamic_lookup,-flat_namespace -Wl,-undefined -Wl,suppress") in
src/tools/enzyme/enzyme/Enzyme/CMakeLists.txt:155

if (APPLE)
# Darwin-specific linker flags for loadable modules.
set_target_properties(LLVMEnzyme-${LLVM_VERSION_MAJOR} PROPERTIES
    LINK_FLAGS "-dynamiclib -Wl,-undefined,dynamic_lookup,-flat_namespace -Wl,-undefined -Wl,suppress")
if (${Clang_FOUND})
set_target_properties(ClangEnzyme-${LLVM_VERSION_MAJOR} PROPERTIES
    LINK_FLAGS "-dynamiclib -Wl,-undefined,dynamic_lookup,-flat_namespace -Wl,-undefined -Wl,suppress")
endif()
set_target_properties(LLDEnzyme-${LLVM_VERSION_MAJOR} PROPERTIES
    LINK_FLAGS "-dynamiclib -Wl,-undefined,dynamic_lookup,-flat_namespace -Wl,-undefined -Wl,suppress")
endif()

results in

[1/57] Linking CXX shared module Enzyme/LLVMEnzyme-17.dylib
FAILED: Enzyme/LLVMEnzyme-17.dylib
: && /usr/bin/c++ -Wall -fno-rtti -ffunction-sections -fdata-sections -fPIC -arch arm64 -stdlib=libc++ -Werror=unused-variable -Werror=dangling-else -Werror=unused-but-set-variable -Werror=return-type -Werror=nonnull -Werror=unused-result -Werror=reorder -Werror=switch -O2 -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk -bundle -Wl,-headerpad_max_install_names -dynamiclib -Wl,-undefined,dynamic_lookup,-flat_namespace -Wl,-undefined -Wl,suppress -o Enzyme/LLVMEnzyme-17.dylib Enzyme/CMakeFiles/LLVMEnzyme-17.dir/ActivityAnalysis.cpp.o Enzyme/CMakeFiles/LLVMEnzyme-17.dir/ActivityAnalysisPrinter.cpp.o Enzyme/CMakeFiles/LLVMEnzyme-17.dir/CApi.cpp.o Enzyme/CMakeFiles/LLVMEnzyme-17.dir/CacheUtility.cpp.o Enzyme/CMakeFiles/LLVMEnzyme-17.dir/CallDerivatives.cpp.o Enzyme/CMakeFiles/LLVMEnzyme-17.dir/DiffeGradientUtils.cpp.o Enzyme/CMakeFiles/LLVMEnzyme-17.dir/DifferentialUseAnalysis.cpp.o Enzyme/CMakeFiles/LLVMEnzyme-17.dir/Enzyme.cpp.o Enzyme/CMakeFiles/LLVMEnzyme-17.dir/EnzymeLogic.cpp.o Enzyme/CMakeFiles/LLVMEnzyme-17.dir/FunctionUtils.cpp.o Enzyme/CMakeFiles/LLVMEnzyme-17.dir/GradientUtils.cpp.o Enzyme/CMakeFiles/LLVMEnzyme-17.dir/InstructionBatcher.cpp.o Enzyme/CMakeFiles/LLVMEnzyme-17.dir/JLInstSimplify.cpp.o Enzyme/CMakeFiles/LLVMEnzyme-17.dir/MustExitScalarEvolution.cpp.o Enzyme/CMakeFiles/LLVMEnzyme-17.dir/PreserveNVVM.cpp.o Enzyme/CMakeFiles/LLVMEnzyme-17.dir/TraceGenerator.cpp.o Enzyme/CMakeFiles/LLVMEnzyme-17.dir/TraceInterface.cpp.o Enzyme/CMakeFiles/LLVMEnzyme-17.dir/TraceUtils.cpp.o Enzyme/CMakeFiles/LLVMEnzyme-17.dir/Utils.cpp.o Enzyme/CMakeFiles/LLVMEnzyme-17.dir/SCEV/ScalarEvolutionExpander.cpp.o Enzyme/CMakeFiles/LLVMEnzyme-17.dir/TypeAnalysis/TypeTree.cpp.o Enzyme/CMakeFiles/LLVMEnzyme-17.dir/TypeAnalysis/TypeAnalysis.cpp.o Enzyme/CMakeFiles/LLVMEnzyme-17.dir/TypeAnalysis/TypeAnalysisPrinter.cpp.o Enzyme/CMakeFiles/LLVMEnzyme-17.dir/TypeAnalysis/RustDebugInfo.cpp.o  -Wl,-rpath,@loader_path/../lib -Wl,-rpath,/Users/marc/rust/build/aarch64-apple-darwin/llvm/lib  /Users/marc/rust/build/aarch64-apple-darwin/llvm/lib/libLLVM.dylib && :
clang: error: invalid argument '-bundle' not allowed with '-dynamiclib'

@ZuseZ4
Copy link
Member

ZuseZ4 commented Aug 1, 2024

Looking through old PRs and online docs, you can apparently get rid of -bundle flags by changing the CMake build type from Module to something else, but that caused a lot of compilation failures in CI, so not sure if its' the right path to proceed: EnzymeAD/Enzyme#1683

@marcpabst
Copy link
Author

marcpabst commented Aug 1, 2024

I managed to get it to build (I'll try to re-trace my steps later)! I then ran

rustup toolchain link enzyme build/host/stage1
rustup toolchain install nightly

But now I'm stuck with

~/rust-autodiff-test % cargo +enzyme run -r
   Compiling rust-autodiff-test v0.1.0 (/Users/marc/rust-autodiff-test)
error: the `-Z unstable-options` flag must also be passed to enable the flag `check-cfg`

Adding -Z unstable-options doesn't change the error message.

@marcpabst
Copy link
Author

marcpabst commented Aug 1, 2024

RUSTFLAGS="-Z unstable-options" did the trick! Thanks for your help, looks like it's working fine now!

(unrelated, but looks like Zed just got support for rust-toolchain.toml, yay!)

@marcpabst
Copy link
Author

What fixed it in the end was mainly EnzymeAD/Enzyme#1683 plus some file naming shenanigans, so that the compiler can find the right dylib.

@ZuseZ4
Copy link
Member

ZuseZ4 commented Aug 1, 2024

Can you share your final rust changes so we can merge them?

@marcpabst
Copy link
Author

marcpabst commented Aug 1, 2024

Sure!

Hoewever, building using those changes seems to result in a runtime dependency on libLLVM.dylib and libLLVMEnzyme-17.dylib:

   Finished `release` profile [optimized] target(s) in 0.03s
     Running `target/release/rust-autodiff-test`
dyld[58129]: <FED69655-D158-3B3E-B082-72B7E293A37E> /Users/marc/rust-autodiff-test/target/release/rust-autodiff-test
dyld[58129]: <6F889319-3E79-3EA9-AFF4-A7C3BFFCF8F3> /Users/marc/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/libLLVMEnzyme-17.dylib
dyld[58129]: <C9A4714E-2961-3FBD-8E74-8615B3A5777E> /usr/lib/libSystem.B.dylib
dyld[58129]: <B0558732-E048-33CE-A6D7-2EC929BFCA1D> /usr/lib/system/libcache.dylib
dyld[58129]: <1A4B497A-69D2-3048-8F47-133482DABAFA> /usr/lib/system/libcommonCrypto.dylib
dyld[58129]: <DE627B59-A9DD-30C5-8F6D-BEF3949C64A1> /usr/lib/system/libcompiler_rt.dylib
dyld[58129]: <11CC75EB-6597-33C6-B725-607F7E0B0A28> /usr/lib/system/libcopyfile.dylib
dyld[58129]: <61DEF451-18FC-3FA7-B05A-F80B0BF36404> /usr/lib/system/libcorecrypto.dylib
dyld[58129]: <A53D555D-F748-3010-83FE-385C660A81BD> /usr/lib/system/libdispatch.dylib
dyld[58129]: <33108EF5-2DC1-3E6D-B3BE-C55520E1594F> /usr/lib/system/libdyld.dylib
dyld[58129]: <FCB5DECB-A48D-3B82-9E12-03F2AFA3B0F9> /usr/lib/system/libkeymgr.dylib
dyld[58129]: <8C3F607B-5FA9-3C80-AC97-66F2BBCF2F85> /usr/lib/system/libmacho.dylib
dyld[58129]: <9A7AD27E-0ECF-3D69-AD56-4C41E0AC19DE> /usr/lib/system/libquarantine.dylib
dyld[58129]: <DE28FFB5-D896-38E2-B585-E9847A7918C3> /usr/lib/system/libremovefile.dylib
dyld[58129]: <092396DB-281D-3FA0-9E50-43F0B11AA972> /usr/lib/system/libsystem_asl.dylib
dyld[58129]: <41DD58AD-E581-3BC7-A2C4-314046BA471F> /usr/lib/system/libsystem_blocks.dylib
dyld[58129]: <AB8A0A0B-9A1E-3284-9243-5A0334468AE5> /usr/lib/system/libsystem_c.dylib
dyld[58129]: <15F74475-A05D-333E-85B8-8BCFF0170FAC> /usr/lib/system/libsystem_collections.dylib
dyld[58129]: <136CD597-BE32-3806-8C7E-B144FCC956C6> /usr/lib/system/libsystem_configuration.dylib
dyld[58129]: <9176BF94-EF84-3C52-93FD-7014FD23E850> /usr/lib/system/libsystem_containermanager.dylib
dyld[58129]: <9DF87415-44A0-31F3-BC0B-BFBC6D5BFC2B> /usr/lib/system/libsystem_coreservices.dylib
dyld[58129]: <A7805817-0D12-3663-8B9B-EE13AA69A1C4> /usr/lib/system/libsystem_darwin.dylib
dyld[58129]: <458D27F4-5C6F-312C-A552-437FE7B0F705> /usr/lib/system/libsystem_darwindirectory.dylib
dyld[58129]: <4FCBB463-F915-38A4-AB32-159AF74479E9> /usr/lib/system/libsystem_dnssd.dylib
dyld[58129]: <E361D92B-A13D-3278-A67E-059BCC1A1942> /usr/lib/system/libsystem_featureflags.dylib
dyld[58129]: <0B3340BE-7CBE-356F-9ACE-7A062DC0554E> /usr/lib/system/libsystem_info.dylib
dyld[58129]: <DBAEE968-94BC-390A-9277-2F0D6696F804> /usr/lib/system/libsystem_m.dylib
dyld[58129]: <690A8B04-8E64-3332-B5A5-56A3D5C1C43F> /usr/lib/system/libsystem_malloc.dylib
dyld[58129]: <DB1022B7-FDD1-3045-A287-3149C1B3D7DA> /usr/lib/system/libsystem_networkextension.dylib
dyld[58129]: <60ECA172-D5DA-3743-AE08-2E397A8C7754> /usr/lib/system/libsystem_notify.dylib
dyld[58129]: <5F138857-5FE4-30BE-8B63-FDD645DF23A1> /usr/lib/system/libsystem_sandbox.dylib
dyld[58129]: <C2EA7F93-A263-39A2-BCDC-F26DCE9DFA32> /usr/lib/system/libsystem_secinit.dylib
dyld[58129]: <CA94FC21-BC40-3B43-B65D-B87ECE9E1D48> /usr/lib/system/libsystem_kernel.dylib
dyld[58129]: <E1840DAA-9C10-393D-A3A3-94DDE6FE0011> /usr/lib/system/libsystem_platform.dylib
dyld[58129]: <A7D94C96-7B1F-3229-9BEA-048D037C3292> /usr/lib/system/libsystem_pthread.dylib
dyld[58129]: <16E4B671-D97C-3908-9127-307A83DF3DCD> /usr/lib/system/libsystem_symptoms.dylib
dyld[58129]: <1F45FD2F-3ABC-3AB3-9A6F-CC9C727ACBDA> /usr/lib/system/libsystem_trace.dylib
dyld[58129]: <0C443A31-8851-30E2-9FAD-B4C7633F5135> /usr/lib/system/libunwind.dylib
dyld[58129]: <272DC84F-1871-3ADD-8126-2155D6EFF7AB> /usr/lib/system/libxpc.dylib
dyld[58129]: <E812122A-DA5B-3F1A-A15C-936C0B9D26BE> /usr/lib/libc++abi.dylib
dyld[58129]: <9BAB9556-7A2A-30A8-ACDE-010BA8E2367D> /usr/lib/libobjc.A.dylib
dyld[58129]: <B4164258-BFB1-3871-A2E2-528C373AEF77> /usr/lib/liboah.dylib
dyld[58129]: <375254EB-5A65-35D0-B6B4-773E6763D289> /usr/lib/libc++.1.dylib
dyld[58129]: <3EDF0285-F0A0-3AD6-83E2-F4A7AAE38709> /Users/marc/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/libLLVM.dylib
dyld[58129]: <0E96113D-04E8-3404-A96A-8D5DE355F9B8> /usr/lib/libz.1.dylib
y = 10
dy = 7

@ZuseZ4
Copy link
Member

ZuseZ4 commented Aug 1, 2024

But only of rustc which I'd expect, but not of your binary, right? I.e. if you run cargo +enzyme build and look at the resulting binary, it won't depend on the two, right?

@marcpabst
Copy link
Author

marcpabst commented Aug 1, 2024

No, looks its weirdly an actual dependency of the binary: (this is with export DYLD_PRINT_LIBRARIES=1)

(base) marc@eduroam-int-dhcp-97-103-145 rust-autodiff-test % ./target/release/rust-autodiff-test
dyld[58776]: <FED69655-D158-3B3E-B082-72B7E293A37E> /Users/marc/rust-autodiff-test/target/release/rust-autodiff-test
dyld[58776]: Library not loaded: @rpath/libLLVMEnzyme-17.dylib
  Referenced from: <FED69655-D158-3B3E-B082-72B7E293A37E> /Users/marc/rust-autodiff-test/target/release/rust-autodiff-test
  Reason: no LC_RPATH's found
zsh: abort      ./target/release/rust-autodiff-test

@ZuseZ4
Copy link
Member

ZuseZ4 commented Aug 1, 2024

That's bad, but by default (if Enzyme isn't moved between compiling and running the binary) is going to work, right?
So I'd still merge the fix and then try to remove the dependency next.

cc @tgymnich since we just talked about it.

@marcpabst
Copy link
Author

marcpabst commented Aug 1, 2024

Well, cargo run works, but calling the binary directly results in the error above (I assume cargo sets some environment variables that make it work?) Edit: look like cargo indeed sets DYLD_FALLBACK_LIBRARY_PATH.

I only noticed this because I tried to build and import a Python module using Pyo3 and Python's importlib gives a helpful error when it can't locate a dynaimc library.

@ZuseZ4
Copy link
Member

ZuseZ4 commented Aug 2, 2024

I just merged a different fix from Tim, @marcpabst can you try to rebuild and see if A) it works and B) if you still have the runtime dependency of your resulting test/example binaries on LLVM and Enzyme?

@marcpabst
Copy link
Author

This is what I'm seeing:

[10/11] Install the project...
-- Install configuration: "Release"
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: no LC_RPATH load command with path: /Users/marc/tmo/rust/build/aarch64-apple-darwin/llvm/lib found in: /Users/marc/tmo/rust/build/aarch64-apple-darwin/enzyme/lib/libEnzyme-17.dylib (for architecture arm64), required for specified option "-delete_rpath /Users/marc/tmo/rust/build/aarch64-apple-darwin/llvm/lib"
-- Installing: /Users/marc/tmo/rust/build/aarch64-apple-darwin/enzyme/lib/libEnzymeBCLoad-17.dylib
cargo:root=/Users/marc/tmo/rust/build/aarch64-apple-darwin/enzyme
	finished in 5.101 seconds
thread 'main' panicked at src/lib.rs:1605:24:
src.symlink_metadata() failed with No such file or directory (os error 2)
stack backtrace:
   0: rust_begin_unwind
             at /rustc/0e09125c6c3c2fd70d7de961bcf0e51575235fad/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/0e09125c6c3c2fd70d7de961bcf0e51575235fad/library/core/src/panicking.rs:72:14
   2: bootstrap::Build::copy_internal
             at ./src/bootstrap/src/lib.rs:1605:24
   3: bootstrap::Build::copy
             at ./src/bootstrap/src/lib.rs:1593:9
   4: <bootstrap::core::build_steps::compile::Assemble as bootstrap::core::builder::Step>::run
             at ./src/bootstrap/src/core/build_steps/compile.rs:1657:13
   5: bootstrap::core::builder::Builder::ensure
             at ./src/bootstrap/src/core/builder.rs:2192:23
   6: bootstrap::core::builder::Builder::compiler
             at ./src/bootstrap/src/core/builder.rs:989:9
   7: <bootstrap::core::build_steps::compile::Std as bootstrap::core::builder::Step>::make_run
             at ./src/bootstrap/src/core/build_steps/compile.rs:107:23
   8: bootstrap::core::builder::StepDescription::maybe_run
             at ./src/bootstrap/src/core/builder.rs:328:13
   9: bootstrap::core::builder::StepDescription::run
             at ./src/bootstrap/src/core/builder.rs:398:17
  10: bootstrap::core::builder::Builder::run_step_descriptions
             at ./src/bootstrap/src/core/builder.rs:981:9
  11: bootstrap::core::builder::Builder::execute_cli
             at ./src/bootstrap/src/core/builder.rs:962:9
  12: bootstrap::Build::build
             at ./src/bootstrap/src/lib.rs:672:13
  13: bootstrap::main
             at ./src/bootstrap/src/bin/main.rs:85:5
  14: core::ops::function::FnOnce::call_once
             at /rustc/0e09125c6c3c2fd70d7de961bcf0e51575235fad/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Build completed unsuccessfully in 0:00:14

@ZuseZ4
Copy link
Member

ZuseZ4 commented Aug 2, 2024

Can you share what file it's looking for, and how your Enzyme build folder looks like?

@marcpabst
Copy link
Author

It's looking for src = "/Users/marc/tmo/rust/build/aarch64-apple-darwin/enzyme/build/Enzyme/libEnzyme-17.so".

(base) marc@Marcs-MBP-2 rust % ls /Users/marc/tmo/rust/build/aarch64-apple-darwin/enzyme/build/Enzyme/
BinopDerivatives.inc		BlasDerivatives.inc		BlasTA.inc			CallDerivatives.inc.d		InstructionDerivatives.inc.d	LLVMEnzyme-17.dylib
BinopDerivatives.inc.d		BlasDerivatives.inc.d		BlasTA.inc.d			Clang				IntrinsicDerivatives.inc	cmake_install.cmake
BlasAttributor.inc		BlasDiffUse.inc			CMakeFiles			ClangEnzyme-17.dylib		IntrinsicDerivatives.inc.d	libEnzyme-17.dylib
BlasAttributor.inc.d		BlasDiffUse.inc.d		CallDerivatives.inc		InstructionDerivatives.inc	LLDEnzyme-17.dylib

@ZuseZ4
Copy link
Member

ZuseZ4 commented Aug 2, 2024

Ah good, only the file ending is wrong (so vs dylib), but at least the libenzyme version exists. Do you want to quickly test a fix and make a PR if you find that we're under macos?
cc @tgymnich, did your Enzyme build with .so endings, or why did it work for you locally?

@tgymnich
Copy link
Member

tgymnich commented Aug 2, 2024

@ZuseZ4 maybe I overlooked some error. It should use .dylib on mac. Possibly also .dll on windows?

@marcpabst
Copy link
Author

Some quick changes in src/bootstrap/src/core/build_steps/compile.rs lead to a successful build:

  if let Some(enzyme_install) = enzyme_install {
      let src_lib = enzyme_install.join("build/Enzyme/libEnzyme-17.so");

      let libdir = builder.sysroot_libdir(build_compiler, build_compiler.host);
      let target_libdir = builder.sysroot_libdir(target_compiler, target_compiler.host);
      let dst_lib = libdir.join("libEnzyme-17.so");
      let target_dst_lib = target_libdir.join("libEnzyme-17.so");
      builder.copy(&src_lib, &dst_lib);
      builder.copy(&src_lib, &target_dst_lib);
  }
if let Some(enzyme_install) = enzyme_install {
    let src_lib = enzyme_install.join("build/Enzyme/libEnzyme-17.dylib");

    let libdir = builder.sysroot_libdir(build_compiler, build_compiler.host);
    let target_libdir = builder.sysroot_libdir(target_compiler, target_compiler.host);
    let dst_lib = libdir.join("libEnzyme-17.dylib");
    let target_dst_lib = target_libdir.join("libEnzyme-17.dylib");
    builder.copy(&src_lib, &dst_lib);
    builder.copy(&src_lib, &target_dst_lib);
}

@ZuseZ4
Copy link
Member

ZuseZ4 commented Aug 2, 2024

Windows is unfortunately unsupported anyway by Enzyme core, since MSVC dislikes Enzyme's interesting
#declare private public

ok, then I can make a quick PR.

@marcpabst
Copy link
Author

Unfortunately, looks like the runtime dependency issue persists - here's the otool -l output:

./target/release/rust-autodiff-test:
Load command 0
      cmd LC_SEGMENT_64
  cmdsize 72
  segname __PAGEZERO
   vmaddr 0x0000000000000000
   vmsize 0x0000000100000000
  fileoff 0
 filesize 0
  maxprot 0x00000000
 initprot 0x00000000
   nsects 0
    flags 0x0
Load command 1
      cmd LC_SEGMENT_64
  cmdsize 552
  segname __TEXT
   vmaddr 0x0000000100000000
   vmsize 0x0000000000048000
  fileoff 0
 filesize 294912
  maxprot 0x00000005
 initprot 0x00000005
   nsects 6
    flags 0x0
Section
  sectname __text
   segname __TEXT
      addr 0x00000001000044e0
      size 0x0000000000037e88
    offset 17632
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x80000400
 reserved1 0
 reserved2 0
Section
  sectname __stubs
   segname __TEXT
      addr 0x000000010003c368
      size 0x0000000000000318
    offset 246632
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x80000408
 reserved1 0 (index into indirect symbol table)
 reserved2 12 (size of stubs)
Section
  sectname __const
   segname __TEXT
      addr 0x000000010003c680
      size 0x000000000000427c
    offset 247424
     align 2^4 (16)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __gcc_except_tab
   segname __TEXT
      addr 0x00000001000408fc
      size 0x0000000000001260
    offset 264444
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __unwind_info
   segname __TEXT
      addr 0x0000000100041b5c
      size 0x0000000000000ef8
    offset 269148
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __eh_frame
   segname __TEXT
      addr 0x0000000100042a58
      size 0x00000000000055a8
    offset 272984
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x6800000b
 reserved1 0
 reserved2 0
Load command 2
      cmd LC_SEGMENT_64
  cmdsize 232
  segname __DATA_CONST
   vmaddr 0x0000000100048000
   vmsize 0x0000000000004000
  fileoff 294912
 filesize 16384
  maxprot 0x00000003
 initprot 0x00000003
   nsects 2
    flags 0x10
Section
  sectname __got
   segname __DATA_CONST
      addr 0x0000000100048000
      size 0x0000000000000218
    offset 294912
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000006
 reserved1 66 (index into indirect symbol table)
 reserved2 0
Section
  sectname __const
   segname __DATA_CONST
      addr 0x0000000100048218
      size 0x0000000000002038
    offset 295448
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Load command 3
      cmd LC_SEGMENT_64
  cmdsize 472
  segname __DATA
   vmaddr 0x000000010004c000
   vmsize 0x0000000000004000
  fileoff 311296
 filesize 16384
  maxprot 0x00000003
 initprot 0x00000003
   nsects 5
    flags 0x0
Section
  sectname __data
   segname __DATA
      addr 0x000000010004c000
      size 0x0000000000000038
    offset 311296
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __thread_vars
   segname __DATA
      addr 0x000000010004c038
      size 0x00000000000000a8
    offset 311352
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000013
 reserved1 0
 reserved2 0
Section
  sectname __thread_data
   segname __DATA
      addr 0x000000010004c0e0
      size 0x0000000000000020
    offset 311520
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000011
 reserved1 0
 reserved2 0
Section
  sectname __thread_bss
   segname __DATA
      addr 0x000000010004c100
      size 0x0000000000000032
    offset 311552
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000012
 reserved1 0
 reserved2 0
Section
  sectname __bss
   segname __DATA
      addr 0x000000010004c138
      size 0x00000000000000c0
    offset 0
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000001
 reserved1 0
 reserved2 0
Load command 4
      cmd LC_SEGMENT_64
  cmdsize 72
  segname __LINKEDIT
   vmaddr 0x0000000100050000
   vmsize 0x0000000000014000
  fileoff 327680
 filesize 71136
  maxprot 0x00000001
 initprot 0x00000001
   nsects 0
    flags 0x0
Load command 5
      cmd LC_DYLD_CHAINED_FIXUPS
  cmdsize 16
  dataoff 327680
 datasize 1456
Load command 6
      cmd LC_DYLD_EXPORTS_TRIE
  cmdsize 16
  dataoff 329136
 datasize 80
Load command 7
     cmd LC_SYMTAB
 cmdsize 24
  symoff 329912
   nsyms 766
  stroff 342704
 strsize 52856
Load command 8
            cmd LC_DYSYMTAB
        cmdsize 80
      ilocalsym 0
      nlocalsym 696
     iextdefsym 696
     nextdefsym 3
      iundefsym 699
      nundefsym 67
         tocoff 0
           ntoc 0
      modtaboff 0
        nmodtab 0
   extrefsymoff 0
    nextrefsyms 0
 indirectsymoff 342168
  nindirectsyms 133
      extreloff 0
        nextrel 0
      locreloff 0
        nlocrel 0
Load command 9
          cmd LC_LOAD_DYLINKER
      cmdsize 32
         name /usr/lib/dyld (offset 12)
Load command 10
     cmd LC_UUID
 cmdsize 24
    uuid B9820BE7-FF08-373D-B415-2713C18CD8DB
Load command 11
      cmd LC_BUILD_VERSION
  cmdsize 32
 platform 1
    minos 14.0
      sdk 14.2
   ntools 1
     tool 3
  version 1022.1
Load command 12
      cmd LC_SOURCE_VERSION
  cmdsize 16
  version 0.0
Load command 13
       cmd LC_MAIN
   cmdsize 24
  entryoff 18196
 stacksize 0
Load command 14
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name /usr/lib/libSystem.B.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 1336.61.1
compatibility version 1.0.0
Load command 15
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name @rpath/libEnzyme-17.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 0.0.0
compatibility version 0.0.0
Load command 16
      cmd LC_FUNCTION_STARTS
  cmdsize 16
  dataoff 329216
 datasize 696
Load command 17
      cmd LC_DATA_IN_CODE
  cmdsize 16
  dataoff 329912
 datasize 0
Load command 18
      cmd LC_CODE_SIGNATURE
  cmdsize 16
  dataoff 395568
 datasize 3248

@wsmoses
Copy link
Member

wsmoses commented Aug 2, 2024

Windows is unfortunately unsupported anyway by Enzyme core, since MSVC dislikes Enzyme's interesting #declare private public

ok, then I can make a quick PR.

Enzyme is supported on Windows -- you just have to use clang to build not msvc

@ZuseZ4
Copy link
Member

ZuseZ4 commented Aug 2, 2024

Agree there are workarounds including also e.g. WSL2, but as long as the default Rust x86_64-pc-windows-msvc toolchain doesn't work due to this Enzyme core issue, I won't go around and claim win support, because for the default setup of the majority that just wouldn't be true.

@marcpabst
Copy link
Author

marcpabst commented Aug 7, 2024

If this is an MSVC-related issue, does the x86_64-pc-windows-gnu toolschain then work fine on windows?

@ZuseZ4
Copy link
Member

ZuseZ4 commented Aug 7, 2024

windows-gnu should work (iirc. someone confirmed it in another issue here), but please feel free to test and confirm that if you have a machine at hand.

@ZuseZ4
Copy link
Member

ZuseZ4 commented Aug 15, 2024

@marcpabst Can you test if the dependency still exists on the latest master? Since I rebased, you probably want to delete the whole folder and clone it again. https://enzyme.mit.edu/index.fcgi/rust/installation.html

@marcpabst
Copy link
Author

Hey, I've been having a bit of trouble cloning the repo - will give it another try tonight!

@marcpabst
Copy link
Author

Something is still wrong withc cloning the repos:

Updating submodule src/tools/enzyme
Submodule 'src/tools/enzyme' (git@github.com:EnzymeAD/Enzyme.git) registered for path 'src/tools/enzyme'
Cloning into '/Users/marc/rust/src/tools/enzyme'...
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@github.com:EnzymeAD/Enzyme.git' into submodule path '/Users/marc/rust/src/tools/enzyme' failed
Failed to clone 'src/tools/enzyme'. Retry scheduled
Cloning into '/Users/marc/rust/src/tools/enzyme'...
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@github.com:EnzymeAD/Enzyme.git' into submodule path '/Users/marc/rust/src/tools/enzyme' failed
Failed to clone 'src/tools/enzyme' a second time, aborting
Cloning into '/Users/marc/rust/src/tools/enzyme'...
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@github.com:EnzymeAD/Enzyme.git' into submodule path '/Users/marc/rust/src/tools/enzyme' failed
Failed to clone 'src/tools/enzyme'. Retry scheduled
Cloning into '/Users/marc/rust/src/tools/enzyme'...
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@github.com:EnzymeAD/Enzyme.git' into submodule path '/Users/marc/rust/src/tools/enzyme' failed
Failed to clone 'src/tools/enzyme' a second time, aborting
submodule src/tools/enzyme does not appear to be checked out, but it is required for this step
The Enzyme sources are required for autodiff.
Build completed unsuccessfully in 0:00:42

@ZuseZ4
Copy link
Member

ZuseZ4 commented Aug 21, 2024

Maybe try to use https, like I do it in this PR?
https://github.com/rust-lang/rust/pull/129176/files#diff-fe7afb5c9c916e521401d3fcfb4277d5071798c3baf83baf11d6071742823584
Let me know if this works.

@wsmoses
Copy link
Member

wsmoses commented Aug 21, 2024

@ZuseZ4 The error is here

url = git@github.com:EnzymeAD/Enzyme.git
this should be changed to https like the others in the .gitmodules

@ZuseZ4
Copy link
Member

ZuseZ4 commented Aug 21, 2024

That's the link I posted ^^
But thanks. Out of curiosity, why? Is that for people who haven't configured a working keychain for authentication?
I had it as git@github.com... for months if not years and it used to work for others.
The reason why I changed it during upstreaming is that Rust CI also didn't work with the git link:

Edit: Nevermind, my bad: https://github.com/EnzymeAD/rust/pull/166/files#diff-fe7afb5c9c916e521401d3fcfb4277d5071798c3baf83baf11d6071742823584
Apparently, I had it correct in the last iteration too, and just got it wrong during re-implementation. Keeping 80+ files with rustc up to date is challenging, I can't wait for the first ones to merge upstream.

@marcpabst
Copy link
Author

Thanks, I'll give it another go later!

@jedbrown
Copy link

I think this issue is settled, but just noting that git clone --recursive should work out of the box even for people who don't have GitHub accounts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants