XGBoost4j missing dylib for apple sillicon m1

[martintreurnicht]

CPU: Apple M1 Max
RAM: 64GB

java -version:

  openjdk version "11.0.13" 2021-10-19 LTS
  OpenJDK Runtime Environment Zulu11.52+13-CA (build 11.0.13+8-LTS)
  OpenJDK 64-Bit Server VM Zulu11.52+13-CA (build 11.0.13+8-LTS, mixed mode)

LIBRARY: ml.dmlc:xgboost4j_2.12:1.5.1

Getting the following error when trying to run our model tests.

java.lang.ExceptionInInitializerError
	at ml.dmlc.xgboost4j.java.Booster.init(Booster.java:694)
	at ml.dmlc.xgboost4j.java.Booster.<init>(Booster.java:51)
	at ml.dmlc.xgboost4j.java.Booster.loadModel(Booster.java:80)
	at ml.dmlc.xgboost4j.java.XGBoost.loadModel(XGBoost.java:77)

Seems like the root cause at the bottom of the stacktrace is a missing dylib:

Caused by: java.io.FileNotFoundException: File /lib/macos/aarch64/libxgboost4j.dylib was not found inside JAR.
	at ml.dmlc.xgboost4j.java.NativeLibLoader.createTempFileFromResource(NativeLibLoader.java:233)
	at ml.dmlc.xgboost4j.java.NativeLibLoader.loadLibraryFromJar(NativeLibLoader.java:176)
	at ml.dmlc.xgboost4j.java.NativeLibLoader.initXGBoost(NativeLibLoader.java:130)
	at ml.dmlc.xgboost4j.java.XGBoostJNI.<clinit>(XGBoostJNI.java:34)
	... 88 more

Is there a known fix for this?

[hcho3]

Currently XGBoost4J does not support Apple Silicon. Sorry for inconvenience.

[martintreurnicht]

Currently XGBoost4J does not support Apple Silicon. Sorry for inconvenience.

Got it, can you provide some kind of timeline here?

[hcho3]

@martintreurnicht We don't know yet. None of us (XGBoost developers) own an M1 machine, and currently I am not aware of any CI service that lets you test Apple Silicon.

[Craigacp]

You can compile xgboost4j from source on an M1 machine and it works. I did this last year when they came out on my personal M1 MBP, but without build resources to make the binaries it's hard to deploy.

This PR I wrote from January modifies the loader so you can package the aarch64 dylib in to a jar file containing an x86 one (and the other OS binaries as well) and everything works, so you can push that to an internal artifactory if you have one.

[martintreurnicht]

@Craigacp Ok awesome, thanks! Yeah I got it to build and works great. I now just need to combine this dylib with the existing jar. This will unblock us for now

[martintreurnicht]

@martintreurnicht We don't know yet. None of us (XGBoost developers) own an M1 machine, and currently I am not aware of any CI service that lets you test Apple Silicon.

@hcho3 I see you guys use github actions, you can probably use something like https://www.macstadium.com/mac-mini. Setup self-hosted with mac stadium should do the trick

[hcho3]

Thanks for the pointer. We'll revisit once we have enough bandwidth and funding (since we'd have to pay MacStadium)

[martintreurnicht]

Ok fully understand, is there no way to cross compile this on x86_64 macOS? I don't know too much about this but i would imagine it must be possible, you can build aarch64 compatible binaries in XCode on a x86_64 mac after all

[Craigacp]

I think cross-compiling the JNI binding will be tricky. You'll probably need to have an ARM64 JVM as well as an x86 one to ensure you have the right libraries to link against.

[trivialfis]

Closing as duplicated to #6408