Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add presets for LZ4 #1094

Merged
merged 10 commits into from
Oct 27, 2021
Merged

Add presets for LZ4 #1094

merged 10 commits into from
Oct 27, 2021

Conversation

HedgehogCode
Copy link
Contributor

This PR adds presets for very fast compression and decompression with lz4.

The presets are currently enabled for Linux, Mac OS, and Windows on x86 and x86_64. It is probably possible to enable them on other platforms, but I haven't had time to look into it yet.

The GitHub Action for Windows is failing, and I neither have a Windows system nor enough experience with C/C++ compilers and linking to fix it. @chaubold started looking into it, but any help from someone experienced with JavaCPP presets would be welcome.

@saudet
Copy link
Member

saudet commented Oct 25, 2021

Out of curiosity, what's missing from https://github.com/lz4/lz4-java?

@HedgehogCode
Copy link
Contributor Author

https://github.com/lz4/lz4-java does not support the LZ4 Frame Format sufficiently.

  • "Dependent Blocks" in the Frame Format are not supported.
  • There is only an implementation based on streams which is inefficient if you already have the data in memory as you need it (like with Apache Arrow ArrowBuf).

The goal is to use these presets for the Java implementation of Apache Arrow. Currently, Apache Commons Compress is used because of the limitations with lz4-java and the Frame format. However, this is very slow.

@saudet saudet merged commit a70841e into bytedeco:master Oct 27, 2021
@HedgehogCode HedgehogCode deleted the lz4 branch October 27, 2021 09:24
@HedgehogCode
Copy link
Contributor Author

Thank you for the fixes and for merging this so quickly!
May I ask if there is a release schedule for JavaCPP? We (at KNIME) would like to know if we can wait for the release and use the released version or if we should use a pre-release version for now.

@saudet
Copy link
Member

saudet commented Oct 27, 2021 via email

@HedgehogCode
Copy link
Contributor Author

Hi @saudet, trying the snapshots first was a good suggestion. We discovered problems on Mac OS:

java.lang.UnsatisfiedLinkError: no jnilz4 in java.library.path: [/Users/jenkins/Library/Java/Extensions, /Library/Java/Extensions, /Network/Library/Java/Extensions, /System/Library/Java/Extensions, /usr/lib/java, .]
	at java.base/java.lang.ClassLoader.loadLibrary(Unknown Source)
	at java.base/java.lang.Runtime.loadLibrary0(Unknown Source)
	at java.base/java.lang.System.loadLibrary(Unknown Source)
	at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1762)
	at org.bytedeco.javacpp.Loader.load(Loader.java:1369)
	at org.bytedeco.javacpp.Loader.load(Loader.java:1181)
	at org.bytedeco.javacpp.Loader.load(Loader.java:1157)
	at org.bytedeco.lz4.global.lz4.<clinit>(lz4.java:14)
	at org.knime.core.columnar.arrow.compress.Lz4BlockCompressionCodec.doDecompress(Lz4BlockCompressionCodec.java:95)
	[...]
Caused by: java.lang.UnsatisfiedLinkError: /Users/jenkins/.javacpp/cache/macosx-x86_64/libjnilz4.dylib: dlopen(/Users/jenkins/.javacpp/cache/macosx-x86_64/libjnilz4.dylib, 1): Library not loaded: /usr/local/lib/liblz4.1.dylib
  Referenced from: /Users/jenkins/.javacpp/cache/macosx-x86_64/libjnilz4.dylib
  Reason: image not found
	at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native Method)
	at java.base/java.lang.ClassLoader$NativeLibrary.load(Unknown Source)
	at java.base/java.lang.ClassLoader$NativeLibrary.loadLibrary(Unknown Source)
	at java.base/java.lang.ClassLoader.loadLibrary0(Unknown Source)
	at java.base/java.lang.ClassLoader.loadLibrary(Unknown Source)
	at java.base/java.lang.Runtime.load0(Unknown Source)
	at java.base/java.lang.System.load(Unknown Source)
	at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1709)
	... 48 more

It seems like the libjnilz4 library can't find the liblz4 library because it only looks at /usr/local/lib/. Could this be related to #46? If you don't have an idea I can try to get access to a Mac and do some debugging.

@saudet
Copy link
Member

saudet commented Nov 3, 2021

Right, I always forget. The rpath is never set correctly on Mac. We can hack it after the build with install_name_tool this way: https://github.com/bytedeco/javacpp-presets/blob/master/mxnet/cppbuild.sh#L213
Could you send a pull request to that effect? You can check that it worked with otool -L ...

@HedgehogCode
Copy link
Contributor Author

Awesome. Does not look too hard. I will take a look. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants