Snappy should use randomize directory or library name when importing .so file

[pwendell]

If two users on the same machine use snappy-java they will end up using the same temp directory (e.g. /tmp) by default. Due to file permissions this can cause a runtime exception if user A creates the file and then user B tries to access the file later but doesn't have permissions.

I see two very simple fixes for this. One is to create a randomized sub-directory of "java.io.tmpdir" when loading. The second is to add a time-stamp or randomized prefix to the .so library.

Technically users can work around this by setting the temp dir explicitly, but the out-of-the-box behavior does seem like it could be improved.

We ran into this issue in Spark:
https://issues.apache.org/jira/browse/SPARK-2881

[xerial]

@pwendell
In snappy-java, the extract .so library name is appended with a random UUID. So this would be a problem of finding a writable directory to place .so files.

Do you have any suggestion on where we should place such a temporary file?

[pwendell]

Hey @xerial - we are using snappy-java 1.0.5 and it doesn't appear to have a UUID. Maybe this was fixed in a more recent version? If so, then we can just close the issue.

$ ls /tmp/ |grep snappy
snappy-1.0.5-libsnappyjava.so

[xerial]

Yes. Please use snappy-java-1.1.1.3.jar, the latest version. Its API is compatible with 1.0.5 series.

[pwendell]

Okay - we'll just upgrade then, but we'll probably wait to do this until after the 1.1 release which is frozen a this point. Thanks - sorry for the false issue.

[xerial]

No problem.
Please let me know if upgrading snappy-java introduces another problem :)

[xerial]

@pwendell
I just deployed a hot fix that appends UUID to the extracted lib file name as 1.0.5.3 release, which will be available in Maven central in a few hours.

[xerial]

I recommend to use 1.1.1.3, which is fast and has a more reliable native code loading mechanism, but upgrading to 1.0.5.3 would be less harm since it just adds this hot fix and powerPC support to 1.0.5.

[pwendell]

Hey Taro,

Thanks for backporting that fix. I think we will go ahead and use the
1.0.5.3 release for Spark 1.1. In Spark's master branch we will upgrade to
1.1.1.3. I had a workaround prepared for Spark 1.1, but it depended on the
order of static initialization which is very brittle.

https://github.com/apache/spark/pull/1994/files

• Patrick

On Sat, Aug 16, 2014 at 8:22 PM, Taro L. Saito notifications@github.com
wrote:

I recommend to use 1.1.1.3, which is fast and has a more reliable native
code loading mechanism, but upgrading to 1.0.5.3 would be less harm since
it just adds this hot fix and powerPC support to 1.0.5.

Reply to this email directly or view it on GitHub
#84 (comment).