CRIU criu_pingPerf_testCreateImageAndUnprivilegedRestore_0_FAILED with The JVM attempted to load libcriu.so but was unable to: 1 on zLinux

[llxia]

Failure link

internal/Grinder
machine: fyrlx11l.z.svl.ibm.com
TARGET=criu_pingPerf_testCreateImageAndUnprivilegedRestore

00:01:02.904  java version "11.0.19" 2023-04-18
00:01:02.904  IBM Semeru Runtime Certified Edition 11.0.19+4 (build 11.0.19+4)
00:01:02.904  Eclipse OpenJ9 VM 11.0.19+4 (build master-ec981665a, JRE 11 Linux amd64-64-Bit Compressed References 20230304_560 (JIT enabled, AOT enabled)
00:01:02.904  OpenJ9   - ec981665a
00:01:02.904  OMR      - d5af8d16e
00:01:02.904  JCL      - 66af94dab5 based on jdk-11.0.19+4)

00:19:16.817  Launching defaultServer (Open Liberty 23.0.0.3-beta/wlp-1.0.74.cl230220230222-1257) on Eclipse OpenJ9 VM, version 11.0.19-internal+0-adhoc.jenkins.BuildJDK11s390xlinuxcriuNightly (en_US)
00:19:16.817  CWWKE0953W: This version of Open Liberty is an unsupported early release version.
00:19:16.817  [AUDIT   ] CWWKE0001I: The server defaultServer has been launched.
00:19:16.817  [AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/checkpoint.xml
00:19:16.817  [AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/keystore.xml
00:19:16.818  [AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/open-default-port.xml
00:19:16.818  [AUDIT   ] CWWKZ0058I: Monitoring dropins for applications.
00:19:16.818  [WARNING ] CWOWB1009W: Implicit bean archives are disabled.
00:19:17.794  [AUDIT   ] CWWKT0016I: Web application available (default_host): http://ed7800485acd:9080/pingperf/
00:19:17.794  [AUDIT   ] CWWKZ0001I: Application pingperf started in 0.600 seconds.
00:19:17.794  [AUDIT   ] CWWKC0451I: A server checkpoint was requested. When the checkpoint completes, the server stops.
00:19:17.794  [ERROR   ] CWWKC0453E: The server checkpoint request failed with the following message: The JVM attempted to load libcriu.so but was unable to: 1
00:19:17.794  [AUDIT   ] CWWKF0012I: The server installed the following features: [cdi-3.0, checkpoint-1.0, concurrent-2.0, jndi-1.0, jsonp-2.0, restfulWS-3.0, restfulWSClient-3.0, servlet-5.0].
00:19:17.795  [AUDIT   ] CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 1.587 seconds.
00:19:17.795  [AUDIT   ] CWWKE0084I: The server defaultServer is stopping because thread Checkpoint failed, exiting... (00000048) called the method java.lang.System.exit: 
00:19:17.795  	at java.base/java.lang.System.exit(System.java:495)
00:19:17.795  	at io.openliberty.checkpoint.internal.CheckpointImpl.lambda$checkpointOrExitOnFailure$1(CheckpointImpl.java:265)
00:19:17.795  	at java.base/java.lang.Thread.run(Thread.java:839)
00:19:17.795  
00:19:17.795  [AUDIT   ] CWWKE1100I: Waiting for up to 30 seconds for the server to quiesce.
00:19:17.795  [AUDIT   ] CWWKT0017I: Web application removed (default_host): http://ed7800485acd:9080/pingperf/
00:19:18.786  -----------------------------------
00:19:18.786  criu_pingPerf_testCreateImageAndUnprivilegedRestore_0_FAILED

Optional info

This test creates criu test image and runs restore on the same machine.
The same test passed on xlinux.

[VermaSh]

It's failing because libcriu.so is missing.

...
00:19:17.794  [ERROR   ] CWWKC0453E: The server checkpoint request failed with the following message: The JVM attempted to load libcriu.so but was unable to: 1
...

Comparing the CRIU builds for x and zLinux I found that the library is located in ./usr/local/lib64/libcriu.so for x and in ./usr/local/lib/libcriu.so for zLinux. So that might be the reason for the failure.

[1] https://github.com/ibmruntimes/semeru-containers/blob/0c9ed3568c5db2498cea1ddd88d8c5d61ab02148/17-ea/jdk/ubi/ubi8/Dockerfile.open.releases.full#L46-L47 "x criu build"
[2] https://na-public.artifactory.swg-devops.com/artifactory/sys-rt-generic-local/hyc-runtimes-jenkins.swg-devops.com/build-scripts/criu_build/12/criu.tar.gz "zLinux criu build"

[llxia]

Just to clarify, in pingPerf test, we use the docker file from Liberty https://raw.githubusercontent.com/OpenLiberty/ci.docker/instanton/releases/latest/beta-instanton/Dockerfile.ubi.openjdk17, not from semeru container file or criu_build.

[VermaSh]

I don't think they have zLinux binaries

> curl -LO https://public.dhe.ibm.com/ibmdl/export/pub/software/openliberty/runtime/criu-build/b6/criu
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 7138k  100 7138k    0     0  3438k      0  0:00:02  0:00:02 --:--:-- 3438k

> chmod u+x criu

> ./criu --version
-bash: ./criu: cannot execute binary file: Exec format error

[llxia]

Close it as we updated the workflow to use dockerfile in https://github.com/ibmruntimes/semeru-containers