[BUG] Opensearch worker unable to start in docker: SIGILL

[baileywickham]

Describe the bug

I'm having an issue starting a new opensearch instance in docker locally. I'm on a brand new Macbook M4 pro. This error shows up using the default docker virtualization settings, I've tried using Docker VVM and toggling Rosetta virtualization but that didn't help. I suspect this might be a docker specific issue because I am able to get Opensearch running using the arm binary natively outside of docker. I'm also not able to repro this issue on a M2 Macbook Air.

Command:

docker run -d \                                                                                                                                                        
  -e "DISABLE_PERFORMANCE_ANALYZER=true" \
  -p 9200:9200 -p 9600:9600 \
  opensearchproject/opensearch:latest

Stack trace:

2024-12-02 09:53:33 # A fatal error has been detected by the Java Runtime Environment:
2024-12-02 09:53:33 #
2024-12-02 09:53:33 #  SIGILL (0x4) at pc=0x0000ffff7bd3fc5c, pid=15, tid=16
2024-12-02 09:53:33 #
2024-12-02 09:53:33 # JRE version:  (21.0.5+11) (build )
2024-12-02 09:53:33 # Java VM: OpenJDK 64-Bit Server VM (21.0.5+11-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-aarch64)
2024-12-02 09:53:33 # Problematic frame:
2024-12-02 09:53:33 # j  java.lang.System.registerNatives()V+0 java.base@21.0.5
2024-12-02 09:53:33 #
2024-12-02 09:53:33 # No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
2024-12-02 09:53:33 #
2024-12-02 09:53:33 # An error report file with more information is saved as:
2024-12-02 09:53:33 # /usr/share/opensearch/hs_err_pid15.log
2024-12-02 09:53:33 [0.022s][warning][os] Loading hsdis library failed
2024-12-02 09:53:33 #
2024-12-02 09:53:33 # The crash happened outside the Java Virtual Machine in native code.
2024-12-02 09:53:33 # See problematic frame for where to report the bug.
2024-12-02 09:53:33 #
2024-12-02 09:23:50 /usr/share/opensearch/plugins/opensearch-security/tools/install_demo_configuration.sh: line 64:    15 Aborted                 "$JAVA" -Dorg.apache.logging.log4j.simplelog.StatusLogger.level=OFF -cp "$DIR/../*:$DIR/../../../lib/*:$DIR/../deps/*" org.opensearch.security.tools.democonfig.Installer "$DIR" "$@" 2> /dev/null
2024-12-02 09:53:33 /usr/share/opensearch/plugins/opensearch-security/tools/install_demo_configuration.sh: line 64:    15 Aborted                 "$JAVA" -Dorg.apache.logging.log4j.simplelog.StatusLogger.level=OFF -cp "$DIR/../*:$DIR/../../../lib/*:$DIR/../deps/*" org.opensearch.security.tools.democonfig.Installer "$DIR" "$@" 2> /dev/null

uname -a

Darwin Baileys-MacBook-Pro.local 24.2.0 Darwin Kernel Version 24.2.0: Fri Nov 15 18:56:44 PST 2024; root:xnu-11215.61.2.501.1~1/RELEASE_ARM64_T6041 arm64

I do not have issues starting the opensearch dashboard in a separate container.

I'm not really sure how to go about tracking this down, any help would be appreciated. Let me know what additional info I can provide.

Related component

Build

To Reproduce

1. Install Docker
2. Run

docker run -d \                                                                                                                                                        
  -e "DISABLE_PERFORMANCE_ANALYZER=true" \
  -p 9200:9200 -p 9600:9600 \
  opensearchproject/opensearch:latest

Expected behavior

Opensearch server/worker start successfully.

Additional Details

Plugins
Defaults

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: OSX
• Version: Opensearch latest

Additional context
Add any other context about the problem here.

[zoltan-kollar]

Possibly related: corretto-21/issues/85

[baileywickham]

That looks related, yes. I'm on MacOS 15.2 (24C101) so that is likely the issue.

[zoltan-kollar]

I've tried to apply -XX:UseSVE=0 as the suggested workaround by passing it to OPENSEARCH_JAVA_OPTS env varriable, but it did not work for me. Did you get any luck with it?

This is the command I tried:

docker run -it \
  -e "DISABLE_SECURITY_PLUGIN=true" \
  -e "discovery.type=single-node" \
  -e "OPENSEARCH_JAVA_OPTS=-XX:UseSVE=0" \
  opensearchproject/opensearch:latest

[tedstevko-warner]

Found the same issue on my M4 Mac with 15.2, and tried the -XX:UseSVE=0 workaround.

It looks like there's a Java program "org.opensearch.tools.java_version_checker.JavaVersionChecker" which is being run during image startup, but without "OPENSEARCH_JAVA_OPTS" or "JAVA_OPTS" in the command line. So it runs Java without the "-XX:UseSVE=0" prop for that program call, causing the docker image to fail with a SIGILL and therefore the image fails to load.

It traces back to this bug: https://bugs.openjdk.org/browse/JDK-8345296 in the JDK.

JavaVersionChecker and all java calls during image startup probably need to be examined to see if they can take in either JAVA_OPTS or OPENSEARCH_JAVA_OPTS so anyone running a M4 Mac with 15.2 can add the prop and get around this. Or the latest version of the docker image needs to be updated with a newer version of OpenJDK to take in the fix.