xtdb.rocksdb on ARM Mac

[eneroth]

I'm getting this error when attempting to follow the "in-a-box" example:

Execution error (UnsatisfiedLinkError) at java.lang.ClassLoader$NativeLibrary/load0 (ClassLoader.java:-2).
/private/var/folders/sn/69qx269x1y785_jjlgfvhc2c0000gn/T/crux_rocksdb-6.12.7/librocksdbjni-osx.jnilib: dlopen(/private/var/folders/sn/69qx269x1y785_jjlgfvhc2c0000gn/T/crux_rocksdb-6.12.7/librocksdbjni-osx.jnilib, 1): no suitable image found.  Did find:
	/private/var/folders/sn/69qx269x1y785_jjlgfvhc2c0000gn/T/crux_rocksdb-6.12.7/librocksdbjni-osx.jnilib: mach-o, but wrong architecture
	/private/var/folders/sn/69qx269x1y785_jjlgfvhc2c0000gn/T/crux_rocksdb-6.12.7/librocksdbjni-osx.jnilib: mach-o, but wrong architecture

Is M1 unsupported by Crux, or will I need to employ some trick?

[jarohen]

Hi @eneroth, thanks for raising this 🙏 M1's unfortunately as yet unsupported by RocksDB, see facebook/rocksdb#7720. We have had some success internally under Rosetta - cc @armincerf and @refset.

James

[armincerf]

I would recommend using something like sdkman which provides a simple interface for switching JDK versions. You can also install your JDK manually under Rosetta, but it can be a bit of a pain to switch back to native if you do it yourself. This guide should help https://itnext.io/how-to-install-x86-and-arm-jdks-on-the-mac-m1-apple-silicon-using-sdkman-872a5adc050d?gi=8aa0b090e71e

[eneroth]

It seems so close to being done as well :) It looks like they have RocksDB itself running fine on M1, and are now just missing the Java bindings. I'll subscribe to the thread to keep updated on the progress.

@armincerf Cheers, I use jenv for this, but it's the same idea I think. It feels bad to settle for merely excellent performance once you've got used to outstanding performance though :)

[armincerf]

@eneroth yep that works too. And agreed about performance, I have to wait a whole extra 10 seconds for my repl to start, although I do like to remind myself that even under rosetta its still 15 seconds less than my old XPS used to take 😄 We'll update here too once the issue is fixed in Crux.

[eneroth]

Excellent is good, but outstanding is excellent. 👍

[deobald]

In the interest of helping our M1 users, since the resolution isn't necessarily obvious from the above thread:

As of this post, this problem still exists. RocksDB runs on ARM Macs, but the Java bindings do not. The current workaround is to use sdkman or equivalent to run XTDB using an Intel JDK under Rosetta.

[avocade]

So close now…

facebook/rocksdb#7720 (comment)

[ariskk]

I can't explain why, but sdkman +

sdk install java 11.0.13.fx-zulu

did the trick for me.

[eneroth]

sdk install java 11.0.13.fx-zulu

Huh, interesting. I tried it and it just crashed with:

.sdkman/candidates/java/11.0.13.fx-zulu/zulu-11.jdk/Contents/Home/bin/java is loading libcrypto in an unsafe way

[deobald]

You can try switching from libgcrypt to OpenSSL or SHA1. For example:

XTDB_ENABLE_BYTEUTILS_SHA1=true

This sets a flag here:

xtdb/core/src/xtdb/hash.clj

Lines 10 to 12 in 013cdfd

	(def ^:const ^:private gcrypt-enabled? (not (Boolean/parseBoolean (System/getenv "XTDB_DISABLE_LIBGCRYPT"))))
	(def ^:const ^:private openssl-enabled? (not (Boolean/parseBoolean (System/getenv "XTDB_DISABLE_LIBCRYPTO"))))
	(def ^:const ^:private byte-utils-sha1-enabled? (Boolean/parseBoolean (System/getenv "XTDB_ENABLE_BYTEUTILS_SHA1")))

...not ideal, but it does work around this problem for recent MacOS releases and/or recent JDK releases, which sometimes surface this java is loading libcrypto in an unsafe way error.

[eneroth]

Got it to work, thanks 👍

However, when I check out the Activity Monitor, it's still running as an Intel process, which leads me to believe that sdkman has downloaded the Intel build, regardless.

I manually downloaded the ARM build from Azul's website, and tried with that, and it gives the same error as previously.

It's crazy how much faster it is than the Intel version, though.

[chadangelelli]

I got it to work with XTDB 1.20/Java 8 on 2020 M1 Macbook Air with this:

1. Install sdkman
2. Edit ~/.sdkman/etc/config and change sdkman_rosetta2_compatible=false to sdkman_rosetta2_compatible=true
3. Call sdk install java 8.0.312.fx-zulu (the point here is this is not ARM64)
4. Call sdk use java 8.0.312.fx-zulu (if you didn't select yes at the prompt)
5. Edit ~/.zshrc and add the line export XTDB_ENABLE_BYTEUTILS_SHA1=true
6. source ~/.zshrc or open a new Terminal/iTerm to enable changes
7. ... and back to Clojure (clj/nREPL/whatever)

$ ./dev/repl.sh
nrepl server at port 40000
[Rebel readline] Type :repl/help for online help info
user=> (require '[xtdb-test :as db])
nil
user=> (db/start-db!)
2021-12-27T01:42:56.253Z m INFO [xtdb.tx:326] - Started tx-ingester
#'xtdb-test.db/xtdb-node
user=> db/xtdb-node
#<XtdbNode>
user=>

NOTES:

• i'm guessing more than version 8 works but don't have time to test right now
• for a super fast test, i ran the following:

(ns xtdb-test
  (:require [clojure.java.io :as io]
            [xtdb.api :as xt]))

(declare xtdb-node)

(defn start-xtdb! []
  (letfn [(kv-store [dir]
            {:kv-store {:xtdb/module 'xtdb.rocksdb/->kv-store
                        :db-dir (io/file dir)
                        :sync? true}})]
    (xt/start-node
     {:xtdb/tx-log (kv-store "data/dev/tx-log")
      :xtdb/document-store (kv-store "data/dev/doc-store")
      :xtdb/index-store (kv-store "data/dev/index-store")})))

(defn start-db!
  []
  (def xtdb-node (start-xtdb!)))

(defn stop-xtdb! []
  (.close xtdb-node))

[avocade]

Looks like RocksDB has merged in the fix for facebook/rocksdb#7720 now…! 🚀

[eneroth]

The good news is that it should be easy to update XTDB when it lands.

As it turns out, switching to an Arm JDK and,

brew install cmake
git clone https://github.com/facebook/rocksdb
cd rocksdb
make rocksdbjavastaticosx_ub
mkdir <path-to-project>/rocksdbjni
cp java/target/rocksdbjni-7.0.0-osx.jar <path-to-project>/rocksdbjni/

Adding,

{:aliases {:macos {:extra-deps {org.rocksdb/rocksdbjni {:local/root "rocksdbjni/rocksdbjni-7.0.0-osx.jar"}}}}}

And calling clojure -M:macos works just as intended. Starting the project is ~3x faster than on Rosetta 2 (Electric Boogaloo).

[avocade]

The good news is that it should be easy to update XTDB when it lands.

Indeed, works great thanks :) The speedup is palpable.

[refset]

Thanks for sharing the snippet @eneroth - just be aware that the 7.0.0 API introduces some breaking API changes which @armincerf has observed to trip up XT's eviction behaviour, in particular due to facebook/rocksdb@c39a808

We will endeavour to add proper support for 7.0.0 once it available via Maven Central.

[eneroth]

Yeah, @refset, bad idea to compile from the main branch rather than the 6.9.3 (or whatever the relase target is for M1 support is).

Since we're only using it for local dev, and it seems to work so far, it's not a big issue. I'll recompile if weirdness starts to happen.

[eneroth]

@jarohen @deobald @refset @armincerf @avocade Good news! Replaced my home baked dep with…

{org.rocksdb/rocksdbjni {:mvn/version "6.29.4.1"}}

… and it seems to work well! So, hopefully that should enable support of M1 without complecting it with the upgrade to RocksDB 7.

I don't quite understand Maven. Supposedly, bindings for 7.0.3 are also out, but I can see neither listed on the page. Nevertheless, they download just fine.

Announcement here.

[jarohen]

Cheers for the heads-up, @eneroth 🙏 have included this version bump in 1.21.0-beta2 which was released this morning.

James

[eneroth]

@jarohen I'm getting Index version on disk: 18 does not match index version of code: 20 upon using beta2.

[jarohen]

@eneroth: we've had to bump the index version in 1.21.0-beta1 to fix a bug, and keep some more statistics to feed the query planner - this will require dropping your index store and re-building it from the transaction log and document store. Let us know if you run into any issues 🙂

James

[whmountains]

This is exciting news! I didn't realize ARM support was so close. I tried to run in Rosetta, but in addition to the horrible REPL startup times there were a number of other issues to which I see fixes in this article! I wish I had found it earlier :)

One stopgap solution which has worked quite well for me is to use Xodus as the KV store instead of RocksDB. LMDB seems to have similar issues running on ARM, so I ruled it out. Xodus is probably slower due to not being native code, but I'll definitely take a much faster REPL startup in exchange for slightly slower queries.

I had to fork the Xodus backend to make it work with the latest XTDB, but other than that it's been working great. I published my fork to Clojars as [org.clojars.cwh/crux-xodus "2.0.0"] if anyone is interested in using it. The namespace within the project is still avisi.xtdb.xodus, so the examples in the documentation should all work as-is.