Problems with predictions on MacBook Air with M1 chip in Java project based on Maven

[pwittchen]

I have the following problem:

I'm using ONNX runtime in version 1.8.1 in the Java project with Maven.

When I invoke:

final OrtEnvironment environment = OrtEnvironment.getEnvironment();

then I get an error:

java.lang.NoClassDefFoundError: Could not initialize class ai.onnxruntime.OrtEnvironment

and I cannot perform prediction.

It works fine on Linux on different distros, but I have problems with executing it on MacBook Air with M1 chip.

[Craigacp]

You need to compile ORT from source on the M1 to get it to work in Java, the jars on Maven Central don't have a macOS ARM64 binary in the jar. It should compile without issue provided you have a native macOS ARM64 JVM (and compiler toolchain). Once you've done that you can copy the resulting dylib into the appropriate directory inside the jar file to make a multiplatform jar. Unfortunately there aren't build resources available to do that for the published Java binaries automatically.

[pwittchen]

Ok. Thanks for the reply @Craigacp.

I'm running ONNX runtime on the Linux server, but I wanted to be able to execute tests on the MacBook Air M1 too for the development purposes.

I have 2 more questions:

• In case I'd like to compile it on my own, which directory is "appropriate"? I mean, where do I need to copy resulting dylib you mentioned?
• Do you have any plans to release a *.jar artifact for macOS with M1 chip to the central repository, so I could simply add it to my build and parametrize it depending on the operating system on which it's being compiled? Or maybe do you have any plans to make existing artifact multiplatform?

Regards,
Piotr

[Craigacp]

If you build ORT from source with the Java API enabled then you'll get a jar which has the native library packaged into it inside the ai/onnxruntime/native/osx-arm64 directory. If you merge that directory with the existing ai/onnxruntime/native/ directory in the jar you can download from Maven Central (which contains osx-x64, win-x64 and linux-x64 folders) you'll end up with a jar that runs on all 4 platforms. You can use the jar tool which comes with the JDK to manipulate the jars.

I've not investigated cross-compiling to produce fat binaries for JNI bindings, though it may be possible (facebook/rocksdb#7720 (comment)). I think the Python binaries are cross-compiled, though I've not checked this. If we do want to make fat binaries for macOS then the JNI loader will need some revision as it currently separates out osx-arm64 and osx-x64.

[snnn]

I think the Python binaries are cross-compiled, though I've not checked this

No. Python's setup.py doesn't support cross-compiling. Though it's possible to cross-compile the C/C++ part then use a separated machine to run the packaging part, it's costly to maintain.

[Craigacp]

I think the Python binaries are cross-compiled, though I've not checked this

No. Python's setup.py doesn't support cross-compiling. Though it's possible to cross-compile the C/C++ part then use a separated machine to run the packaging part, it's costly to maintain.

Ah ok. I was a bit confused by this section in the 1.10 release notes: Added Mac M1 Universal2 build support for a single binary that runs natively on both Apple silicon and Intel-based Macs. These are included in the official packages and can also be built using "-arch arm64 -arch x86_64". What packages do have universal binaries?

[snnn]

onnxruntime-osx-universal2-1.10.0.tgz in our release page.

For mac and python, we could provide one package for x86_64 and another one for universal2. Both can be built on x86_64.

[Craigacp]

onnxruntime-osx-universal2-1.10.0.tgz in our release page.

For mac and python, we could provide one package for x86_64 and another one for universal2. Both can be built on x86_64.

If the universal binary is not much bigger and we can get it to work on the JVM then it might be better to replace the x86_64 macOS binary in the jar with a universal one (after suitable modification of the Java native loader). However further down in that RocksDB thread there's a note saying enabling it completely broke their Mac JVM build though, so I don't think it's necessarily straightforward.

[snnn]

@pwittchen , as a quick fix to unblock you, you download onnxruntime arm binaries from https://github.com/microsoft/onnxruntime/releases/download/v1.10.0/onnxruntime-osx-arm64-1.10.0.tgz, then unzip it, extract the so file, put it in your local jar file inside the ai/onnxruntime/native/osx-arm64 directory.

[Craigacp]

@pwittchen , as a quick fix to unblock you, you download onnxruntime arm binaries from https://github.com/microsoft/onnxruntime/releases/download/v1.10.0/onnxruntime-osx-arm64-1.10.0.tgz, then unzip it, extract the so file, put it in your local jar file inside the ai/onnxruntime/native/osx-arm64 directory.

That won't include the JNI wrapper libonnxruntime4j_jni.dylib which lets Java access the native library.

[snnn]

Ahh... Good point!

[pwittchen]

Thanks for your replies. I'll try to prepare custom jar for M1 chip later. In addition, I'll appreciate dedicated official jar in the Maven Central repository if possible.

Regards,
Piotr

[pwittchen]

@Craigacp @snnn I compiled onnxruntime on my Macbook with the command ./build.sh --build_java and in the java/build/libs/onnxruntime-1.11.0.jar inside this *.jar file, in ai/onnxruntime/native/osx-arm64/ directory there are files libonnxruntime.dylib and libonnxruntime4j_jni.dylib, so according to your comments, this *.jar should work with java projects on Macbook with M1 chip. Nevertheless, I haven't tried to add this custom *.jar file to my project yet. I'll check it later.

[snnn]

If you append

--cmake_extra_defines CMAKE_OSX_ARCHITECTURES="arm64;x86_64"

to the command line arguments of build.sh, you will get universal2 binaries.

[pwittchen]

Ok, I will check this out.

[pwittchen]

By the way, is there any chance of publishing official macOS (or universal) jar with ONNX Runtime to the Maven Central Repository, so I could use it instead of compiling and including custom jar inside the project?

[faxu]

By the way, is there any chance of publishing official macOS (or universal) jar with ONNX Runtime to the Maven Central Repository, so I could use it instead of compiling and including custom jar inside the project?

We're planning this for the next release (1.12)

[menacher]

Release 1.12 does not fix this issue, it is still happening.

[Craigacp]

It is fixed in v1.12.1.

[cugurm]

Has anyone built a valid Java Inference ONNXRuntime 1.12.0 for Darwin Aarch64?

I am facing the weird error:

[ 71%] Generating /Users/micuguro/onnxruntime/java/build/libs/onnxruntime.jar
/bin/sh: line 1: 74844 Killed: 9               /Users/micuguro/gradle-7.6/bin/gradle --console=plain clean jar -x test
make[2]: *** [/Users/micuguro/onnxruntime/java/build/libs/onnxruntime.jar] Error 137
make[1]: *** [CMakeFiles/onnxruntime4j.dir/all] Error 2

Thanks!

[Craigacp]

Yes, though I've not done it recently. Is your JVM an x86 one or an ARM one?

[cugurm]

Thanks, Craigacp! Yes, I accidentally used the broken Java.

When I use proper Java, I face the proxy error when running the build.sh --build_java command:

* What went wrong:
Plugin [id: 'com.diffplug.spotless', version: '5.17.0'] was not found in any of the following sources:

- Gradle Core Plugins (plugin is not in 'org.gradle' namespace)
- Plugin Repositories (could not resolve plugin artifact 'com.diffplug.spotless:com.diffplug.spotless.gradle.plugin:5.17.0')
  Searched in the following repositories:
    Gradle Central Plugin Repository

I saw that this problem is handled by turning off the proxy server (link). Unfortunately, I can not turn off the proxy on the server I use to build this jar.

Is there any alternative?
Maybe a download link for valid ONNXRuntime Java Inference Native Darwin AArch64 v1.12.0 (since the one on the maven is invalid)?

Thank you a lot!