Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to spring cloud gcp 5.7.0 netty_internal_tcnative_SSLContext_JNI_OnLoad #3296

Closed
showmethemoney opened this issue Oct 10, 2024 · 6 comments
Labels
priority: p2 type: bug Something isn't working

Comments

@showmethemoney
Copy link

showmethemoney commented Oct 10, 2024

Development environment:

  • Spring Boot version: 3.3.4
  • Docker image: eclipse-temurin:21-jre-alpine

After upgrading the version of spring-cloud-gcp from 5.6.1 to 5.7.0, the application fails to start when running docker run. The error message is as follows:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00000000000204b6, pid=1, tid=7
#
# JRE version: OpenJDK Runtime Environment Temurin-21.0.4+7 (21.0.4+7) (build 21.0.4+7-LTS)
# Java VM: OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (21.0.4+7-LTS, mixed mode, emulated-client, sharing, tiered, compressed oops, compressed class ptrs, serial gc, linux-amd64)
# Problematic frame:
# C  [libio_grpc_netty_shaded_netty_tcnative_linux_x86_645177916680606850614.so+0x2a154]  netty_internal_tcnative_SSLContext_JNI_OnLoad+0x9c4
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid1.log
[1.635s][warning][os] Loading hsdis library failed
#
# If you would like to submit a bug report, please visit:
#   https://github.com/adoptium/adoptium-support/issues
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
@maximilianbauerConrad
Copy link

is this maybe related? googleapis/google-cloud-java#11219

@matthiasheimerl
Copy link

I had the same issue, but I was able to fix it with reverting one other dependency version I'm using this dependency. I reverted to the version 26.47.0 and now it's working.

<groupId>com.google.cloud</groupId>
<artifactId>libraries-bom</artifactId>
<version>${gcp-libraries-bom.version}</version>
<type>pom</type>
<scope>import</scope>

The hint from @maximilianbauerConrad helped to find the solution

@mpeddada1 mpeddada1 added type: bug Something isn't working priority: p2 labels Oct 10, 2024
JaredHatfield added a commit to UnitVectorY-Labs/ServiceAuthCentral that referenced this issue Oct 27, 2024
A separate application was experiencing the following runtime crash when deploying

```
C [libio_grpc_netty_shaded_netty_tcnative_linux_x86_645219058936957356729.so+0x2a154] netty_internal_tcnative_SSLContext_JNI_OnLoad+0x9c4
```

This looks to be related to GoogleCloudPlatform/spring-cloud-gcp#3296
@JoeWang1127
Copy link
Contributor

Please upgrade spring-cloud-gcp to 5.8.0 to fix this issue.

@showmethemoney
Copy link
Author

@JoeWang1127

upgrade pring-cloud-gcp:5.8.0 & eclipse-temurin:21-jre-alpine
Still doesn't work

@JaredHatfield

This comment was marked as outdated.

@JaredHatfield
Copy link

For the benefit of others that may run across the same issue, after digging into the logs more I found the relevant line of code pointed me in the correct direction.

C  [libio_grpc_netty_shaded_netty_tcnative_linux_x86_644283023756439931644.so+0x2a154]  netty_internal_tcnative_SSLContext_JNI_OnLoad+0x9c4

My understanding of the problem is:

  • Alpine Linux uses musl libc, a lightweight C standard library.
  • Many native libraries, including Netty’s tcnative SSL library, are built and tested against glibc.
  • Incompatibility between musl libc and glibc-based native libraries leads to segmentation faults (SIGSEGV).
  • Resulted in JVM crashes when attempting to initialize native SSL components.

The solution for my specific implementation was moving from amazoncorretto:17-alpine-jdk to gcr.io/distroless/java17-debian12.

The closet related issue I could find to this is: grpc/grpc-java#10096 (comment)

Leaving this comment here in hopes that this helps anyone else that is running into a similar issue as it does seem a breaking change was introduced after version 5.6.1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority: p2 type: bug Something isn't working
Projects
None yet
Development

No branches or pull requests

6 participants