rocksdb support on Mac M1

[rwoodard-prog]

Cannot load librocksdbjni with simple starter code on Mac M1.

I know @DevinTDHa and @maziyarpanahi have have worked on the librocksdbjni issue here. Facebook rocksdb team says it should be working here. However, I still get a crash.

Please see maziyarpanahi/spark-nlp-starter#1 (comment) for full details of the problem.

pos_anc download started this may take some time.
Approximate size to download 3.9 MB
Download done! Loading the resource.
glove_100d download started this may take some time.
Approximate size to download 145.3 MB
Download done! Loading the resource.
Exception in thread "main" java.lang.UnsatisfiedLinkError: Can't load library: /var/folders/jj/v8v5hy3d22ndmxj58b_736kw0000gp/T/librocksdbjni2717270241799440176.jnilib
	at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2633)
	at java.base/java.lang.Runtime.load0(Runtime.java:768)
	at java.base/java.lang.System.load(System.java:1837)
	at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:79)
	at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:57)
	at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:69)
	at org.rocksdb.RocksDB.<clinit>(RocksDB.java:38)
	at com.johnsnowlabs.storage.RocksDBConnection.<init>(RocksDBConnection.scala:27)

Steps to Reproduce

1. Crashing on Mac M1 (with appropriate changes to code) maziyarpanahi/spark-nlp-starter#1 (comment)

Your Environment

See #13079 for my env.

[DevinTDHa]

Hello again @rwoodard-prog,

Also thanks again for reporting this.

I was able to reproduce and resolve the issue on my machine.

The problem seems to be a dependency problem between Spark version > 3.1.x and rocksdb.

Why exactly I still have to find out, but in the meantime the issue can be resolved by using a lower version of Spark. In my case I used version 3.1.3 which can be downloaded at the Spark release archives.

Let me know if this resolves the issue you are having.

[rwoodard-prog]

@DevinTDHa Yes, good solution. I downgraded to Spark 3.1.3 and successfully ran the spark-nlp-starter code. Thank you! Closing the issue.

[maziyarpanahi]

If @DevinTDHa you don't mind I am gonna leave this open. I think Apache Spark started using RocksDB itself at some point, so let's make sure we can make 3.2/3.3 releases also compatible.

[DevinTDHa]

I agree, we will need to do some changes regarding this!

And you are right, Spark version 3.2.0 added a StateStore implementation based on RocksDB.

For future reference:
https://spark.apache.org/releases/spark-release-3-2-0.html
https://issues.apache.org/jira/browse/SPARK-34198

[github-actions]

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment or this will be closed in 5 days