JVM Crash when using GCP Secret Manager and Alpine JVM

[pkernevez]

Tested with eclipse-temurin:17.0.2-jre-alpine and eclipse-temurin:17.0.3_7-jre-alpine

Summary

SpringBoot crash at the very beginning of the startup.
The container is running in a GCP K8s.

Steps to reproduce

Start the image in GCP Cluster.
When Spring initialisation try to connect to GCP the JVM Crash.

Expected results

Have the JVM up & running or an error message in case of network/connection issue.

Actual results

Dump Crash file.

exec java -cp backend -Xmx512m -Xlog:gc*=debug,safepoint*=debug,age*=debug:/opt/app/logs/gc-20220524-1739.log -XX:+CrashOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError -XX:ErrorFile=/data/error.log  --add-opens java.base/java.lang=ALL-UNNAMED  -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.managem 
 2022-05-24 17:39:27.956  INFO [IMM,,] 10 --- [main] c.g.c.s.core.DefaultCredentialsProvider  : Default credentials provider for Google Compute Engine.
 2022-05-24 17:39:27.961  INFO [IMM,,] 10 --- [main] c.g.c.s.core.DefaultCredentialsProvider  : Scopes in use by default credentials: [https://www.googleapis.com/auth/cloud-platform]
 #
 # A fatal error has been detected by the Java Runtime Environment:
 #
 #  SIGSEGV (0xb) at pc=0x0000000000003fd6, pid=10, tid=11
 #
 # JRE version: OpenJDK Runtime Environment Temurin-17.0.3+7 (17.0.3+7) (build 17.0.3+7)
 # Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.3+7 (17.0.3+7, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, serial gc, linux-amd64)
 # Problematic frame:
 # C  0x0000000000003fd6
 #
 # Core dump will be written. Default location: /core.%e.10.%t
 #
 # An error report file with more information is saved as:
 # /opt/app/bin/hs_err_pid10.log
 #
 # If you would like to submit a bug report, please visit:
 #https://github.com/adoptium/adoptium-support/issues
 # The crash happened outside the Java Virtual Machine in native code.
 # See problematic frame for where to report the bug.
 #

Triaging info

Java version: 17 (eclipse-temurin:17.0.3_7-jre-alpine)

openjdk 17.0.3 2022-04-19
OpenJDK Runtime Environment Temurin-17.0.3+7 (build 17.0.3+7)
OpenJDK 64-Bit Server VM Temurin-17.0.3+7 (build 17.0.3+7, mixed mode, sharing)

What is your operating system and platform?
OS: Alpine
Plateform: K8s, version 1.21.10-gke.2000

How did you install Java?
Directly use the official docker image and add my jars.

Did it work before?
Without GCP Secret manager it works fine.

Did you test with other Java versions?

I can't test with version <17 ( our code is not compatible).
I tried with eclipse-temurin:17.0.3_7-jre ( debian_version, bullseye/sid) and it works fine

error.log

[karianna]

Looks like its a call out to Netty via JNI that causes this. Can you confirm if you have the latest up to date Netty libs?

[pkernevez]

We are using
+- io.grpc:grpc-netty-shaded:jar:1.45.0:runtime
It's not exactly the latest one but close to it.
It's the dependency of
com.google.cloud:google-cloud-secretmanager:jar:2.1.4:compile

I also opened an issue here
grpc/grpc-java#8751

It seems related to libc usage.

[karianna]

Yeah they need to provide a musl compatible lib.

[pkernevez]

Solved by using the last version of netty (embedded with the last grpc version):

        <dependency>
            <groupId>io.grpc</groupId>
            <artifactId>grpc-netty-shaded</artifactId>
            <version>1.48.1</version>
            <scope>runtime</scope>
        </dependency>