[BUG] OpenSearch 2.15.0 Ec2DiscoveryPlugin failed to get instance metadata

[nikhanj30]

Describe the bug

Launched OpenSearch 2.15.0 cluster on Amazon Linux 2023 OS and the cluster did not start.

Below are the errors:
[2024-07-16T12:54:24,347][ERROR][o.o.d.e.Ec2DiscoveryPlugin] [hostname] failed to get metadata for [placement/availability-zone]
java.io.IOException: Server returned HTTP response code: 401 for URL: http://169.254.169.254/latest/meta-data/placement/availability-zone
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1998) ~[?:?]
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1599) ~[?:?]
at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
at org.opensearch.discovery.ec2.SocketAccess.doPrivilegedIOException(SocketAccess.java:70) ~[discovery-ec2-2.15.0.jar:2.15.0]
at org.opensearch.discovery.ec2.Ec2DiscoveryPlugin.getAvailabilityZoneNodeAttributes(Ec2DiscoveryPlugin.java:168) [discovery-ec2-2.15.0.jar:2.15.0]
at org.opensearch.discovery.ec2.Ec2DiscoveryPlugin.additionalSettings(Ec2DiscoveryPlugin.java:135) [discovery-ec2-2.15.0.jar:2.15.0]
at org.opensearch.plugins.PluginsService.updatedSettings(PluginsService.java:244) [opensearch-2.15.0.jar:2.15.0]
at org.opensearch.node.Node.(Node.java:502) [opensearch-2.15.0.jar:2.15.0]
at org.opensearch.node.Node.(Node.java:425) [opensearch-2.15.0.jar:2.15.0]
at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) [opensearch-2.15.0.jar:2.15.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) [opensearch-2.15.0.jar:2.15.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) [opensearch-2.15.0.jar:2.15.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.15.0.jar:2.15.0]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.15.0.jar:2.15.0]
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.15.0.jar:2.15.0]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.15.0.jar:2.15.0]
at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.15.0.jar:2.15.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.15.0.jar:2.15.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.15.0.jar:2.15.0]

[2024-07-16T16:39:12,793][WARN ][o.o.b.Natives ] [hostname] unable to load JNA native support library, native methods
will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/opensearch-12972404803927399925/jna15942360218562026931.tmp: /tmp/opensearch-12972404803927399925/jna15942360218562026931.tmp:
failed to map segment from shared object
at java.base/jdk.internal.loader.NativeLibraries.load(Native Method) ~[?:?]
at java.base/jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:331) ~[?:?]
at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:197) ~[?:?]
at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:139) ~[?:?]
at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2418) ~[?:?]
at java.base/java.lang.Runtime.load0(Runtime.java:852) ~[?:?]
at java.base/java.lang.System.load(System.java:2025) ~[?:?]
at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1045) ~[jna-5.13.0.jar:5.13.0 (b0)]
at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:1015) ~[jna-5.13.0.jar:5.13.0 (b0)]
at com.sun.jna.Native.(Native.java:221) ~[jna-5.13.0.jar:5.13.0 (b0)]
at java.base/java.lang.Class.forName0(Native Method) ~[?:?]
at java.base/java.lang.Class.forName(Class.java:421) ~[?:?]
at java.base/java.lang.Class.forName(Class.java:412) ~[?:?]
at org.opensearch.bootstrap.Natives.(Natives.java:60) [opensearch-2.15.0.jar:2.15.0]
at org.opensearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:123) [opensearch-2.15.0.jar:2.15.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:191) [opensearch-2.15.0.jar:2.15.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) [opensearch-2.15.0.jar:2.15.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) [opensearch-2.15.0.jar:2.15.0]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) [opensearch-2.15.0.jar:2.15.0]
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) [opensearch-2.15.0.jar:2.15.0]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.15.0.jar:2.15.0]
at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-2.15.0.jar:2.15.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) [opensearch-2.15.0.jar:2.15.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) [opensearch-2.15.0.jar:2.15.0]

On Amazon Linux 2023, there are changes to retrieve instance metadata. Need to first retrieve a token before invoking the metadata URL.

TOKEN=$(curl --request PUT "http://169.254.169.254/latest/api/token" --header "X-aws-ec2-metadata-token-ttl-seconds: 3600")

Related component

Plugins

To Reproduce

Launch a cluster using OpenSearch 2.15.0 on Amazon Linux 2023 OS. EC2DiscoveryPlugin will error out.

Expected behavior

Add a fix to include token retrieval for amazon linux 2023 and pass that to subsequent calls to retrieve instance metadata.

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[peternied]

[Triage - attendees 1 2 3]
@nikhanj30 Thanks for creating this issue. Without seeing the full opensearch.log (here is an example issue that includes it #14791) it will be hard to look into this issue. We also need to know how OpenSearch was deployed and the EC2 instance configuration.

It looks like the S3 error isn't critical path, and the JNA error could be because an x64 build is attempted to be run on an ARM architecture machine (*G instance types).

[dblock]

Is this still IMDSv1 or forcing you to use v2 (which is not supported yet, #3040)?

[reta]

May be related to https://forum.opensearch.org/t/opensearch-jnanatives-library-issue/20375

[dblock]

Oh I see the JNA error that precedes it.

[nikhanj30]

Modified to use IMDSv1 and also the jvm.options -Djna.tmpdir. After that I don't see this error.

[dblock]

Modified to use IMDSv1 and also the jvm.options -Djna.tmpdir. After that I don't see this error.

I presume jvm.options -Djna.tmpdir alone didn't solve the problem, right? If so, let's close this as a dup of #3040, or edit it to report the JNA problem on Amazon Linux 2023 OS?

[nikhanj30]

As mentioned in my previous comment, I just changed those two and it worked fine.

[dblock]

As mentioned in my previous comment, I just changed those two and it worked fine.

I am trying to find out whether both were required for it to work fine. Can you try changing the -Djna.tmpdir back, does it still work?