Problems with predictions on MacBook Air with M1 chip in Java project based on Maven (even on onnxruntime 1.11.0 - latest at this moment) - no osx-aarch64 dir in the native runtimes dir

[pwittchen]

Hi,

I reported a bug in #10874 regarding the fact that onnxruntime does not work on MacBook Air with M1 chip in Java project based on Maven. There is also information that this bug is fixed in PR #10981 I upgraded dependency to onnxruntime to version 1.11.0 and I still have exactly the same bug as I reported in that issue regardless of that upgrade.

Calling a function:

final OrtEnvironment environment = OrtEnvironment.getEnvironment();

causes an error:

java.lang.NoClassDefFoundError: Could not initialize class ai.onnxruntime.OrtEnvironment

I see that in project dependencies there is:

/onnxruntime/1.11.0/onnxruntime-1.11.0.jar!/ai/onnxruntime/native/osx-x64

but error occurs anyway

Moreover, when I execute a single test instead of my whole suite, then I get an error:

java.lang.UnsatisfiedLinkError: no onnxruntime in java.library.path

Probably, the source of the bug is here:

onnxruntime/java/src/main/java/ai/onnxruntime/OnnxRuntime.java

Line 103 in 74eaaad

String arch = System.getProperty("os.arch", "generic").toLowerCase(Locale.ENGLISH);

in the method initOsArch(). I debugged it and in my case I see the following params:

• detectedOs = osx
• arch = aarch64

final result of os arch: osx-aarch64

and then in init() method:

libraryDirPathProperty = System.getProperty(ONNXRUNTIME_NATIVE_PATH)

libraryDirPathProperty is null where ONNXRUNTIME_NATIVE_PATH is onnxruntime.native.path

Next, in extractFromResources(String library) method, resourcePath is:

/ai/onnxruntime/native/osx-aarch64/libonnxruntime_providers_shared.dylib

and there is no such path.

In the path ai/onnxruntime/native there are only the following dirs:

• linux-x64
• osx-x64
• win-x64

but there is no osx-aarch64, which is detected as my arch on MacBook Air with M1 chip

[Craigacp]

That PR didn't make it into the 1.11 release, and it turns out the packaging was broken for arm64 targets before it. If you compile 1.11.0 from source on an M1 and then change the path inside the jar file from /ai/onnxruntime/native/osx-arm64 to /ai/onnxruntime/native/osx-aarch64 then it should produce a working jar file. Alternatively builds from master should work without modifying the jar file.

[pwittchen]

Ok, thank you for the quick response @Craigacp . I'll check this fix after the next release or alternatively with jar compiled from the master branch.

[pwittchen]

I could not build onnx runtime for Java on MacBook Air M1 on the latest master branch :-(.
It's strange because a few commits ago it worked on the exactly the same computer.
Nevertheless, maybe one of my local system updates broke this build. I don't really know. It may be related to Cmake, but I'm not familiar with this tool to fix it quickly by myself.

First, I invoked:

./build.sh --clean

Then I invoked:

./build.sh --build_java

and I got the following log:

Synchronizing submodule url for 'cmake/external/SafeInt/safeint'
Synchronizing submodule url for 'cmake/external/cub'
Synchronizing submodule url for 'cmake/external/cxxopts'
Synchronizing submodule url for 'cmake/external/date'
Synchronizing submodule url for 'cmake/external/dlpack'
Synchronizing submodule url for 'cmake/external/eigen'
Synchronizing submodule url for 'cmake/external/emsdk'
Synchronizing submodule url for 'cmake/external/flatbuffers'
Synchronizing submodule url for 'cmake/external/googlebenchmark'
Synchronizing submodule url for 'cmake/external/googletest'
Synchronizing submodule url for 'cmake/external/json'
Synchronizing submodule url for 'cmake/external/libprotobuf-mutator'
Synchronizing submodule url for 'cmake/external/mimalloc'
Synchronizing submodule url for 'cmake/external/mp11'
Synchronizing submodule url for 'cmake/external/nsync'
Synchronizing submodule url for 'cmake/external/onnx'
Synchronizing submodule url for 'cmake/external/onnx/third_party/benchmark'
Synchronizing submodule url for 'cmake/external/onnx/third_party/pybind11'
Synchronizing submodule url for 'cmake/external/onnx-tensorrt'
Synchronizing submodule url for 'cmake/external/onnx-tensorrt/third_party/onnx'
Synchronizing submodule url for 'cmake/external/onnx-tensorrt/third_party/onnx/third_party/benchmark'
Synchronizing submodule url for 'cmake/external/onnx-tensorrt/third_party/onnx/third_party/pybind11'
Synchronizing submodule url for 'cmake/external/onnxruntime-extensions'
Synchronizing submodule url for 'cmake/external/protobuf'
Synchronizing submodule url for 'cmake/external/protobuf/third_party/benchmark'
Synchronizing submodule url for 'cmake/external/protobuf/third_party/googletest'
Synchronizing submodule url for 'cmake/external/pytorch_cpuinfo'
Synchronizing submodule url for 'cmake/external/re2'
Synchronizing submodule url for 'cmake/external/tensorboard'
Synchronizing submodule url for 'cmake/external/wil'
Synchronizing submodule url for 'server/external/spdlog'
-- Found Python: /opt/homebrew/opt/python@3.9/bin/python3.9 (found version "3.9.10") found components: Interpreter 
-- Found Python: /opt/homebrew/opt/python@3.9/bin/python3.9 (found suitable version "3.9.10", minimum required is "3.6") found components: Interpreter 
-- 
-- 3.18.1.0
-- Using the single-header code from /Users/pw/Development/prv/java/onnxruntime/cmake/external/json/single_include/
-- 
-- ******** Summary ********
--   CMake version             : 3.22.3
--   CMake command             : /opt/homebrew/Cellar/cmake/3.22.3/bin/cmake
--   System                    : Darwin
--   C++ compiler              : /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
--   C++ compiler version      : 13.0.0.13000029
--   CXX flags                 :  -ffunction-sections -fdata-sections -fvisibility=hidden -fvisibility-inlines-hidden -fstack-protector-strong -DCPUINFO_SUPPORTED -Wno-deprecated -Wnon-virtual-dtor
--   Build type                : Debug
--   Compile definitions       : EIGEN_MPL2_ONLY;PLATFORM_POSIX;__STDC_FORMAT_MACROS
--   CMAKE_PREFIX_PATH         : 
--   CMAKE_INSTALL_PREFIX      : /usr/local
--   CMAKE_MODULE_PATH         : /Users/pw/Development/prv/java/onnxruntime/cmake/external
-- 
--   ONNX version              : 1.11.0
--   ONNX NAMESPACE            : onnx
--   ONNX_USE_LITE_PROTO       : ON
--   USE_PROTOBUF_SHARED_LIBS  : OFF
--   Protobuf_USE_STATIC_LIBS  : ON
--   ONNX_DISABLE_EXCEPTIONS   : OFF
--   ONNX_WERROR               : OFF
--   ONNX_BUILD_TESTS          : OFF
--   ONNX_BUILD_BENCHMARKS     : OFF
--   ONNXIFI_DUMMY_BACKEND     : OFF
--   ONNXIFI_ENABLE_EXT        : OFF
-- 
--   Protobuf compiler         : 
--   Protobuf includes         : 
--   Protobuf libraries        : 
--   BUILD_ONNX_PYTHON         : OFF
-- Java Build is enabled
-- Using gradle: /Users/pw/.sdkman/candidates/gradle/current/bin/gradle
-- GRADLE_ARGS: --console=plain;cmakeBuild;-DcmakeBuildDir=/Users/pw/Development/prv/java/onnxruntime/build/MacOS/Debug
-- Running Java tests
-- Configuring done
CMake Error in external/protobuf/cmake/CMakeLists.txt:
  Imported target "ZLIB::ZLIB" includes non-existent path

    "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.1.sdk/usr/include"

  in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:

  * The path was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and references files it does not
  provide.



CMake Error in external/protobuf/cmake/CMakeLists.txt:
  Imported target "ZLIB::ZLIB" includes non-existent path

    "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.1.sdk/usr/include"

  in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:

  * The path was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and references files it does not
  provide.



CMake Error in external/protobuf/cmake/CMakeLists.txt:
  Imported target "ZLIB::ZLIB" includes non-existent path

    "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.1.sdk/usr/include"

  in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:

  * The path was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and references files it does not
  provide.



-- Generating done
CMake Generate step failed.  Build files cannot be regenerated correctly.
Traceback (most recent call last):
  File "/Users/pw/Development/prv/java/onnxruntime/tools/ci_build/build.py", line 2453, in <module>
    sys.exit(main())
  File "/Users/pw/Development/prv/java/onnxruntime/tools/ci_build/build.py", line 2353, in main
    generate_build_tree(
  File "/Users/pw/Development/prv/java/onnxruntime/tools/ci_build/build.py", line 1163, in generate_build_tree
    run_subprocess(
  File "/Users/pw/Development/prv/java/onnxruntime/tools/ci_build/build.py", line 665, in run_subprocess
    return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env)
  File "/Users/pw/Development/prv/java/onnxruntime/tools/python/util/run.py", line 42, in run
    completed_process = subprocess.run(
  File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/opt/homebrew/bin/cmake', '/Users/pw/Development/prv/java/onnxruntime/cmake', '-Donnxruntime_RUN_ONNX_TESTS=OFF', '-Donnxruntime_BUILD_WINML_TESTS=ON', '-Donnxruntime_GENERATE_TEST_REPORTS=ON', '-DPython_EXECUTABLE=/opt/homebrew/opt/python@3.9/bin/python3.9', '-DPYTHON_EXECUTABLE=/opt/homebrew/opt/python@3.9/bin/python3.9', '-Donnxruntime_ROCM_VERSION=', '-Donnxruntime_USE_MIMALLOC=OFF', '-Donnxruntime_ENABLE_PYTHON=OFF', '-Donnxruntime_BUILD_CSHARP=OFF', '-Donnxruntime_BUILD_JAVA=ON', '-Donnxruntime_BUILD_NODEJS=OFF', '-Donnxruntime_BUILD_OBJC=OFF', '-Donnxruntime_BUILD_SHARED_LIB=ON', '-Donnxruntime_BUILD_APPLE_FRAMEWORK=OFF', '-Donnxruntime_USE_DNNL=OFF', '-Donnxruntime_DNNL_GPU_RUNTIME=', '-Donnxruntime_DNNL_OPENCL_ROOT=', '-Donnxruntime_USE_NNAPI_BUILTIN=OFF', '-Donnxruntime_USE_RKNPU=OFF', '-Donnxruntime_USE_OPENMP=OFF', '-Donnxruntime_USE_NUPHAR_TVM=OFF', '-Donnxruntime_USE_LLVM=OFF', '-Donnxruntime_ENABLE_MICROSOFT_INTERNAL=OFF', '-Donnxruntime_USE_VITISAI=OFF', '-Donnxruntime_USE_NUPHAR=OFF', '-Donnxruntime_USE_TENSORRT=OFF', '-Donnxruntime_TENSORRT_HOME=', '-Donnxruntime_USE_TVM=OFF', '-Donnxruntime_TVM_CUDA_RUNTIME=OFF', '-Donnxruntime_USE_MIGRAPHX=OFF', '-Donnxruntime_MIGRAPHX_HOME=', '-Donnxruntime_CROSS_COMPILING=OFF', '-Donnxruntime_DISABLE_CONTRIB_OPS=OFF', '-Donnxruntime_DISABLE_ML_OPS=OFF', '-Donnxruntime_DISABLE_RTTI=OFF', '-Donnxruntime_DISABLE_EXCEPTIONS=OFF', '-Donnxruntime_MINIMAL_BUILD=OFF', '-Donnxruntime_EXTENDED_MINIMAL_BUILD=OFF', '-Donnxruntime_MINIMAL_BUILD_CUSTOM_OPS=OFF', '-Donnxruntime_REDUCED_OPS_BUILD=OFF', '-Donnxruntime_ENABLE_LANGUAGE_INTEROP_OPS=OFF', '-Donnxruntime_USE_DML=OFF', '-Donnxruntime_USE_WINML=OFF', '-Donnxruntime_BUILD_MS_EXPERIMENTAL_OPS=OFF', '-Donnxruntime_USE_TELEMETRY=OFF', '-Donnxruntime_ENABLE_LTO=OFF', '-Donnxruntime_ENABLE_TRANSFORMERS_TOOL_TEST=OFF', '-Donnxruntime_USE_ACL=OFF', '-Donnxruntime_USE_ACL_1902=OFF', '-Donnxruntime_USE_ACL_1905=OFF', '-Donnxruntime_USE_ACL_1908=OFF', '-Donnxruntime_USE_ACL_2002=OFF', '-Donnxruntime_USE_ARMNN=OFF', '-Donnxruntime_ARMNN_RELU_USE_CPU=ON', '-Donnxruntime_ARMNN_BN_USE_CPU=ON', '-Donnxruntime_ENABLE_NVTX_PROFILE=OFF', '-Donnxruntime_ENABLE_TRAINING=OFF', '-Donnxruntime_ENABLE_TRAINING_OPS=OFF', '-Donnxruntime_ENABLE_TRAINING_TORCH_INTEROP=OFF', '-Donnxruntime_ENABLE_CPU_FP16_OPS=OFF', '-Donnxruntime_USE_NCCL=ON', '-Donnxruntime_BUILD_BENCHMARKS=OFF', '-Donnxruntime_USE_ROCM=OFF', '-Donnxruntime_ROCM_HOME=', '-DOnnxruntime_GCOV_COVERAGE=OFF', '-Donnxruntime_USE_MPI=ON', '-Donnxruntime_ENABLE_MEMORY_PROFILE=OFF', '-Donnxruntime_ENABLE_CUDA_LINE_NUMBER_INFO=OFF', '-Donnxruntime_BUILD_WEBASSEMBLY=OFF', '-Donnxruntime_BUILD_WEBASSEMBLY_STATIC_LIB=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_SIMD=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_EXCEPTION_CATCHING=ON', '-Donnxruntime_ENABLE_WEBASSEMBLY_EXCEPTION_THROWING=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_THREADS=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_DEBUG_INFO=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_PROFILING=OFF', '-Donnxruntime_ENABLE_EAGER_MODE=OFF', '-Donnxruntime_ENABLE_EXTERNAL_CUSTOM_OP_SCHEMAS=OFF', '-Donnxruntime_NVCC_THREADS=1', '-Donnxruntime_ENABLE_CUDA_PROFILING=OFF', '-DCMAKE_OSX_ARCHITECTURES=arm64', '-Donnxruntime_DEV_MODE=ON', '-Donnxruntime_PYBIND_EXPORT_OPSCHEMA=OFF', '-Donnxruntime_ENABLE_MEMLEAK_CHECKER=ON', '-DCMAKE_BUILD_TYPE=Debug']' returned non-zero exit status 1.

[Craigacp]

I usually build it with build.sh --update --build --build_java --config Release. Did you recently upgrade XCode? That can cause weird issues when you need to re-accept various licenses and is usually the source of various build issues for me on macOS.

[pwittchen]

Yes, I upgraded XCode. I'm using it for iOS development and I try to keep it updated. I didn't know it's related to onnxruntime build.

[Craigacp]

ONNX Runtime uses the compilers & system headers it installs, but if it's working for iOS development it should have enabled everything. Try completely deleting the build directory and rerunning the build.sh command.

[pwittchen]

I used your params from the previous comment and it's building :-), so I guess this is the correct way to do it. I'll post an update when the build will finish.

[pwittchen]

Ok, it was built on my machine and inside the jar there is a path: ai/onnxruntime/native/osx-aarch64/, so I guess now it's correct.

Do you plan to release this version to the Maven Central repository in the nearest days?
It'd be helpful for project development on Mac.

[Craigacp]

I think it'll land in the next release whenever that comes, but 1.11.0 will not be updated.

[pwittchen]

I meant a new version - not updating the current one, but ok, I'll wait for the release.

[Craigacp]

I meant a new version - not updating the current one, but ok, I'll wait for the release.

I believe ORT is on a quarterly release schedule for feature releases.

@faxu any chance of Java macOS/arm64 binaries (#10981) landing in a 1.11 point release?

[faxu]

I meant a new version - not updating the current one, but ok, I'll wait for the release.

I believe ORT is on a quarterly release schedule for feature releases.

@faxu any chance of Java macOS/arm64 binaries (#10981) landing in a 1.11 point release?

Targeting 1.12 for including this build in the Maven pkg.

[pwittchen]

Thanks for the reply @faxu. Can you tell when can I expect 1.12 release?

[faxu]

Thanks for the reply @faxu. Can you tell when can I expect 1.12 release?

We just released 1.11, so it will be a few months.

[pwittchen]

Ok, thank you for the reply. I'll wait for that. Luckily, running my project with onnxruntime on Mac is not critical issue at the moment.

[lanking520]

@Craigacp Hi Adam, is there anyway we can test out the JNI on Linux AARCH64? Not sure if Java has any nightly build at the moment.

Also +1 for 1.12 releases with Linux/Mac ARM support, looking forward to it :).

[Craigacp]

I don't believe there are nightly builds for Java (or ORT in general), but you should be able to build it yourself on a Linux aarch64 machine. It'll build a jar with just support for that platform, but you can copy in the other platform libraries from the version on Maven Central to make a jar for internal use if you want (make sure you build Linux aarch64 from the right version tag if you want to do this). I don't frequently build it on Linux aarch64, but I build everything on macOS aarch64 last night while working on a feature.

[lanking520]

I don't believe there are nightly builds for Java (or ORT in general), but you should be able to build it yourself on a Linux aarch64 machine. It'll build a jar with just support for that platform, but you can copy in the other platform libraries from the version on Maven Central to make a jar for internal use if you want (make sure you build Linux aarch64 from the right version tag if you want to do this). I don't frequently build it on Linux aarch64, but I build everything on macOS aarch64 last night while working on a feature.

Thanks for your response. Is there any instruction to build for aarch64 linux? Or it is more like following general build guideline and add --build-java flags? Or can I just use the released shared library to generate JNI so directly?

[Craigacp]

I don't believe there are nightly builds for Java (or ORT in general), but you should be able to build it yourself on a Linux aarch64 machine. It'll build a jar with just support for that platform, but you can copy in the other platform libraries from the version on Maven Central to make a jar for internal use if you want (make sure you build Linux aarch64 from the right version tag if you want to do this). I don't frequently build it on Linux aarch64, but I build everything on macOS aarch64 last night while working on a feature.

Thanks for your response. Is there any instruction to build for aarch64 linux? Or it is more like following general build guideline and add --build-java flags? Or can I just used the released shared library to generate JNI so directly?

I build it with this build.sh --update --build --build_java --config Release --parallel, though you might want to add some more execution providers or alter the build options a little depending on your usecase. That command will make a jar file containing the compiled classes and the two native libraries compiled for the local machine.

You can find the exact build command used to make the release binaries in the CI scripts, but those keep moving around so I'm not sure which is the correct one to point at.

I'm currently building it on a Linux aarch64 VM at the moment, but it's taking a little while to get the environment configured as I've not used this VM for ORT stuff before. I'll report back if there was any additional complexity.

[Craigacp]

It built & the tests ran successfully on an Oracle Linux 7 box running on a 4 core Ampere Altra instance. Took a little while to get the machine setup but the build was without incident.

[lanking520]

It built & the tests ran successfully on an Oracle Linux 7 box running on a 4 core Ampere Altra instance. Took a little while to get the machine setup but the build was without incident.

Nice! For now I will just follow the default build options to see if it can work. If build works, I can probably bundle the JNI so files together with released shared library to have some extra optimization.

[ljordan-pea]

I am trying to use onnx java runtime on an m1 as well. I found you could use x86 flavor of java and things will work. I am currently using:

amazon-corretto-15.jdk

I can't speak for the speed or whatnot, but at least my test pass now and I can keep working :) I am not planning on deploying to macOS.

[Craigacp]

I'd be careful running ONNX Runtime under Rosetta on an M1. I'm not sure if the detection in MLAS assumes all x86 CPUs have AVX instructions or not. If it fails the detection and uses an AVX instruction these are not emulated by Rosetta and can cause an illegal instruction error which will crash the JVM if one is executed under it.

[OlegTheCat]

Hi @faxu! As of 1.12 release, I'm getting this error when using Java package on M1 Mac:

java.lang.UnsatisfiedLinkError: no onnxruntime in java.library.path

As far as I understand, M1 binaries have been placed into ai/onnxruntime/native/osx-arm64/ directory. However corresponding Java code is looking for ai/onnxruntime/native/osx-aarch64/ directory (links 1, 2).
Is it something that has been overlooked or am I doing something wrong? Thanks!

[pwittchen]

@faxu I bumped onnxruntime in my project to 1.12.0 and I still get errors like:

no onnxruntime in java.library.path
java.lang.UnsatisfiedLinkError: no onnxruntime in java.library.path
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1875)

or

Could not initialize class ai.onnxruntime.OrtEnvironment
java.lang.NoClassDefFoundError: Could not initialize class ai.onnxruntime.OrtEnvironment

When I use my compiled *.jar file with onnxruntime, which I created earlier for Mac and include it manually in the project for Mac, then it works. I guess something is wrong with the final jar or you need to release separate jars for different operating systems (Windows, Linux, Mac). Currently, In my project I use separate jar file (from Maven Central Repository) for Windows and Linux and another jar file (compiled from the sources) for Mac.

[faxu]

@mszhanyi / @snnn can you take a look at this for Java M1 packaging?

[lanking520]

I just checked the content and found the osx-arm64 instead of osx-aarch64. One of the naming of the binary is wrong. @Craigacp Happy to jump in for a fix if needed.

[lanking520]

The fix is fairly clear: Either add an additional check here https://github.com/microsoft/onnxruntime/blob/master/java/src/main/java/ai/onnxruntime/OnnxRuntime.java#L111 for Mac to rename to arm64, or renaming the packaging in cmake.

[pwittchen]

Thanks for your comments. I created PR here: #12325 according to your suggestions. Please, review it.

[pwittchen]

My PR was closed without merging after discussion because changes need to be done in the other place (inside tools/ci_build/github/linux/java_copy_strip_binary.sh instead of the java code) according to the good practices. The PR no. #12335 should fix the Maven Central packaging issue. In the meantime, *.jar artifact compiled from the source code on the Mac OS X with M1 can be used during development and works fine.

[russellrc-keebo]

If I understand correctly, this should be fixed by #12335

[snnn]

We will publish a patch release to fix this.

[OlegTheCat]

Hi @snnn! Thanks for looking into this! In the newest version (1.12.1) everything works perfect! I'm able to use ONNX Runtime using Java package on M1 Mac.

[pwittchen]

@russellrc-keebo yes, it's fixed by #12335. In the version 1.12.1 this problem no longer occurs.