[BUG] Compatibility issues with the v1.26.0 of Bottlerocket when running Java-based applications

[Veronica4036]

Image I'm using: bottlerocket-aws-k8s-1.31-x86_64-v1.26.0-85f0d68c

What I expected to happen: All nodes using the latest Bottlerocket AMI should be able to run Java-based pods without any issues.

What actually happened: When Karpenter rolled out new nodes using the latest Bottlerocket AMI, all the Java based pods placed in the new nodes are crashing:

kg po -A -owide | grep -v Running | wc -l
125

kg po -A -owide | wc -l
1756

Steps to Reproduce:

1.) Provision new nodes using the latest Bottlerocket AMI for AWS Kubernetes (bottlerocket-aws-k8s-1.31-x86_64-v1.26.0-85f0d68c).
2.) Deploy Java-based pods on the new nodes.
3.) Observe the pods crashing with the above-mentioned errors.

Logs of the pods running in v.1.26.0:

  ~ ❯ kubectl logs wildfly-pod                                                                                                                                             ✘ TSTP  ⎈ BR_Issue  08:23:11
=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /opt/jboss/wildfly

  JAVA: /usr/lib/jvm/java/bin/java

  JAVA_OPTS:  -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true 

=========================================================================

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f1f96488000, 2555904, 1) failed; error='Operation not permitted' (errno=1)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 2555904 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /opt/jboss/hs_err_pid61.log

Logs of the pods running in v.1.25.0:

  ~ ❯ kubectl logs wildfly-pod                                                                                                                                                 ✘ TSTP  ⎈ Elli  08:24:40
=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /opt/jboss/wildfly

  JAVA: /usr/lib/jvm/java/bin/java

  JAVA_OPTS:  -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true  --add-exports=java.desktop/sun.awt=ALL-UNNAMED --add-exports=java.naming/com.sun.jndi.ldap=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.security=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.management/javax.management=ALL-UNNAMED --add-opens=java.naming/javax.naming=ALL-UNNAMED

=========================================================================

02:54:35,496 INFO  [org.jboss.modules] (main) JBoss Modules version 1.12.0.Final
02:54:36,218 INFO  [org.jboss.msc] (main) JBoss MSC version 1.4.13.Final
02:54:36,234 INFO  [org.jboss.threads] (main) JBoss Threads version 2.4.0.Final
02:54:36,456 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: WildFly Full 25.0.0.Final (WildFly Core 17.0.1.Final) starting
02:54:38,271 INFO  [org.wildfly.security] (ServerService Thread Pool -- 26) ELY00001: WildFly Elytron version 1.17.1.Final
02:54:39,918 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)
02:54:40,017 INFO  [org.xnio] (MSC service thread 1-4) XNIO version 3.8.4.Final
02:54:40,037 INFO  [org.xnio.nio] (MSC service thread 1-4) XNIO NIO Implementation Version 3.8.4.Final
02:54:40,261 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 67) WFLYNAM0001: Activating Naming Subsystem
02:54:40,291 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 54) WFLYCLINF0001: Activating Infinispan subsystem.
02:54:40,341 INFO  [org.jboss.remoting] (MSC service thread 1-3) JBoss Remoting version 5.0.23.Final
02:54:40,316 INFO  [org.jboss.as.webservices] (ServerService Thread Pool -- 76) WFLYWS0002: Activating WebServices Extension
02:54:40,393 INFO  [org.wildfly.extension.microprofile.jwt.smallrye] (ServerService Thread Pool -- 65) WFLYJWT0001: Activating MicroProfile JWT Subsystem
02:54:40,393 INFO  [org.wildfly.extension.microprofile.opentracing] (ServerService Thread Pool -- 66) WFLYTRACEXT0001: Activating MicroProfile OpenTracing Subsystem
02:54:40,389 INFO  [org.wildfly.extension.health] (ServerService Thread Pool -- 53) WFLYHEALTH0001: Activating Base Health Subsystem
02:54:40,297 INFO  [org.jboss.as.jaxrs] (ServerService Thread Pool -- 56) WFLYRS0016: RESTEasy version 4.7.2.Final
02:54:40,385 INFO  [org.wildfly.extension.elytron.oidc._private] (ServerService Thread Pool -- 52) WFLYOIDC0001: Activating WildFly Elytron OIDC Subsystem
02:54:40,354 INFO  [org.jboss.as.jsf] (ServerService Thread Pool -- 61) WFLYJSF0007: Activated the following Jakarta Server Faces Implementations: [main]
02:54:40,389 INFO  [org.wildfly.extension.metrics] (ServerService Thread Pool -- 63) WFLYMETRICS0001: Activating Base Metrics Subsystem
02:54:40,279 WARN  [org.jboss.as.txn] (ServerService Thread Pool -- 74) WFLYTX0013: The node-identifier attribute on the /subsystem=transactions is set to the default value. This is a danger for environments running multiple servers. Please make sure the attribute value is unique.

Workaround: Rolling back the nodes to the previous version v1.25.0 fixed the issue.

[jamt9000]

We are also seeing this with Python libraries: ImportError: /usr/local/lib/python3.12/site-packages/catboost/_catboost.so: cannot make segment writable for relocation: Operation not permitted

[simplyzee]

I'm faced with the same issue as above. For now to temporarily rectify the problem, I've pinned a version of Bottlerocket that Karpenter should use which doesn't have issue.

ec2NodeClass:

spec:
  amiFamily: Bottlerocket
  amiSelectorTerms:
    - name: bottlerocket-aws-k8s-1.30-x86_64-v1.25.0-388e1050

[dtelaroli]

Same error here for java.
nodejs images fails with this error: modprobe: FATAL: Module aufs not found in directory

is it possible disable deploy pre-releases?

[adamnoll]

We are seeing this:

Error occurred during initialization of VM
Failed to mark memory page as executable - check if grsecurity/PaX is enabled

We were on:
"bottlerocket-aws-k8s-1.28-x86_64-v1.26.0-85f0d68c"

Rollback to previous version worked for us as well

[tejisin]

Causing issues for javascript apps as well

<--- Last few GCs --->


<--- JS stacktrace --->


#
# Fatal javascript OOM in MemoryChunk allocation failed during deserialization.

Fix for now by pinning to v1.25 of bottlerocket

spec:
  amiFamily: Bottlerocket
  amiSelectorTerms:
  - id: ami-03d7ba32e17494358

[bcressey]

The 1.26.0 release of Bottlerocket included a change to restrict system services from mapping memory as both writable and executable (bottlerocket-os/bottlerocket-core-kit#158).

Although intended to apply only to the host software, which does not need this capability, the restriction also erroneously applied to applications running inside containers. Software relying on just-in-time (JIT) compilation, such as Java or NodeJS, often needs to mark memory as both writable and executable, and this change caused pods running Java and NodeJS applications to fail.

To mitigate the impact, the 1.26.0 release has been rolled back and 1.25.0 is now marked as latest.

[koooosh]

Closing this issue as the fix for this (referenced above) was released in Bottlerocket v1.26.1: https://github.com/bottlerocket-os/bottlerocket/releases/tag/v1.26.1