jni: libjvm.so or libjawt.so not found at runtime

[robaerd]

Steps to Reproduce the Problem

kdb mount `pwd`/STUDENTS.ini system:/lvas/cm/students ini

#The command kdb mount terminated unsuccessfully with the info:
#Was not able to load such a plugin!

#Maybe you misspelled it, there is no such plugin or the loader has problems.
#You might want to try to set LD_LIBRARY_PATH, use kdb-full or kdb-static.
#Errors/Warnings during loading were:
# Sorry, 11 warnings were issued ;(
#	Sorry, module kdb issued the warning C01200:
#	Installation: Dlopen failed. Could not load module libelektra-jni.so. Reason: libjvm.so: cannot open shared object file: No #such file or directory
#	Mountpoint: /
#	Configfile:
#	At: /home/jenkins/libelektra/src/libs/loader/dl.c:90

This error does occur if the jni plugin is built. This also affects our packages for Ubuntu, Debian and Fedora as jni gets installed as part of the libelektra5-all package.
When setting LD_LIBRARY=$JAVA_HOME/lib/server, then mounting with ini, xml and json is working.
As far as I can tell, the jni plugin shouldn't even be needed for these plugins?

System Information

• Elektra Version: master
• Docker Container with Ubuntu Focal (also occurs on Fedora33)
• OpenJdk11

[markus2330]

Thank you for reporting this problem! This looks quite urgent. This should work by default: is maybe the TOML plugin not available?

This error does occur if the jni plugin is built.

Is there also an error? In what you pasted is only an warning.

As far as I can tell, the jni plugin shouldn't even be needed for these plugins?

Probably not but if the loading of a module fails warnings get emitted anyway.

The easiest fix would be to deploy a /etc/ld.so.conf.d/ file which includes $JAVA_HOME/lib/server. Another approach would be to set RUNPATH like /usr/bin/r is doing. But both does not sound to be the clean way to do it.

@robaerd did you already investigate what is the best way to link against libjvm.so? Maybe we are doing something wrong there. Did you find any other packages which do this? Probably we need to add some CMake magic to make libelektra-jni link against libjvm.so.

[robaerd]

is maybe the TOML plugin not available?

The TOML plugin is available and working.

Is there also an error? In what you pasted is only a warning.

Sorry, no error, but the file does not get mounted.

@robaerd did you already investigate what is the best way to link against libjvm.so?

Not yet, but I can of course do that.

[markus2330]

Sorry, no error, but the file does not get mounted.

Seems like there are even several issues involved here.

Can you give me Docker commands to reproduce the problem?

Not yet, but I can of course do that.

Thank you. It might even be a bug in CMake on these platforms, it looks to me we already link against ${JNI_LIBRARIES}, which should do the job.

https://stackoverflow.com/questions/7715804/how-to-use-find-jni-on-cmake/7716814#7716814

[robaerd]

Can you give me Docker commands to reproduce the problem?

I used the Dockerfile in scripts/docker/ubuntu/focal/Dockerfile and uncommented line 103 USER ${JENKINS_USERID}, so we have root permissions for the kdb mount command.

docker build -t focal-elektra --build-arg JENKINS_USERID=$(id -u) --build-arg JENKINS_GROUPID=$(id -g) -f scripts/docker/ubuntu/focal/Dockerfile scripts/docker/ubuntu/focal/
docker run -it -d --name focal-elektra focal-elektra:latest
docker exec -it focal-elektra /bin/bash

# install Elektra with jni plugin
cd /home/jenkins
git clone https://github.com/ElektraInitiative/libelektra.git
cd libelektra
mkdir build
cd build
cmake -DPLUGINS='ALL;jni' ..
make install

kdb mount test.ini system:/test/ini ini

If you want to reproduce this with the Debian packages on Ubuntu Focal:

docker run -it -d --name focal-elektra-packages ubuntu:focal
docker exec -it focal-elektra-packages /bin/bash

apt update
apt install gnupg software-properties-common apt-transport-https
apt-key adv --keyserver keys.gnupg.net --recv-keys F26BBE02F3C315A19BF1F791A9A25CC1CC83E839
echo "deb https://debs.libelektra.org/focal focal main" | tee /etc/apt/sources.list.d/elektra.list
apt update
apt install libelektra5-all

kdb mount test.ini system:/test/ini ini

[markus2330]

Thank you for the instructions! Would be nice to have such images for the next release!

Part of the problem might be solved with: #3673: I suspect this loop terminated early, not finding the ni plugin. toml actually will not get selected, as it does not provide ini, I changed this now in the private repo.

For JNI loading itself, we also need to find a solution. The linking does not seem to be correct:

ldd /usr/lib/elektra5/libelektra-jni.so 
        ...
        libjvm.so => not found
        ...

[markus2330]

On Debian Buster I get:

> kdb mount `pwd`/STUDENTS.toml system:/lvas/cm/students ini                                                                                                                                         The command kdb mount terminated unsuccessfully with the info:
Was not able to load such a plugin!

Maybe you misspelled it, there is no such plugin or the loader has problems.
You might want to try to set LD_LIBRARY_PATH, use kdb-full or kdb-static.
Errors/Warnings during loading were:
 Sorry, 11 warnings were issued ;(
        Sorry, module kdb issued the warning C01200:
        Installation: Dlopen failed. Could not load module libelektra-jni.so. Reason: libjawt.so: cannot open shared object file: No such file or directory
        Mountpoint: /
        Configfile: 
        At: /home/jenkins/workspace/libelektra-release/libelektra/src/libs/loader/dl.c:90

Please report the issue at https://issues.libelektra.org/

Which seems to be also due to wrong linkage.

[markus2330]

With kdb-spec mount I can reproduce the same issue. Workaround: LD_LIBRARY_PATH=/usr/lib/jvm/java-11-openjdk-amd64/lib/server/:/usr/lib/jvm/java-11-openjdk-amd64/lib

@robaerd On Debian Buster the repo does not seem to contain the latest deb of master? Did you already test my fix #3673

[robaerd]

@robaerd On Debian Buster the repo does not seem to contain the latest deb of master?

Debian Buster repo should contain the latest debs of master. My example above only was for the released packages.
Simply change the line echo "deb https://debs.libelektra.org/focal focal main" | tee /etc/apt/sources.list.d/elektra.list to echo "deb https://debs.libelektra.org/focal focal-unstable main" | tee /etc/apt/sources.list.d/elektra.list.

Did you already test my fix #3673

Yes this fixed the issue where the file did not get mounted. Thank you!

[markus2330]

Thank you, my fault. I forgot the -unstable. So deb https://debs.libelektra.org/buster buster-unstable main does the job.

Yes this fixed the issue where the file did not get mounted. Thank you!

Perfect, so we now back to the problem that loading jni fails. Any new result on that?

[robaerd]

After building Elektra, libjvm.so is found:

$ ldd /home/jenkins/libelektra/build/lib/libelektra-jni.so
	...
	libjvm.so => /usr/lib/jvm/default-java/lib/server/libjvm.so (0x00007f151b479000)
	...

However after installing it can not be found anymore:

$ ldd /usr/local/lib/elektra/libelektra-jni.so
	...
	libjvm.so => not found
	...

I'm not sure why this is the case.

[markus2330]

Yes, I can confirm. make install already breaks it, so it is unrelated to packaging but probably it is related to CMake.

The RUNPATH (readelf) changes from:

 0x000000000000001d (RUNPATH)            Library runpath: [<...>/build/lib:/usr/lib/jvm/default-java/lib:/usr/lib/jvm/default-java/lib/server:]

to:

 0x000000000000001d (RUNPATH)            Library runpath: [/usr/local/lib]

during installation, which might be the problem.

For example, on Debian stretch /usr/bin/r has a RUNPATH:

/usr/bin/r
 0x000000000000001d (RUNPATH)            Library runpath: [/usr/lib/R/lib:/usr/lib/x86_64-linux-gnu:/usr/lib/jvm/default-java/lib/server:/usr/lib/x86_64-linux-gnu/libfakeroot:/usr/lib64/libfakeroot:/usr/lib32/libfakeroot]

So keeping the RUNPATH (or setting an RPATH with CMAKE_INSTALL_RPATH for this module) would be an option how we can solve the problem.