Failed to load credentials from IMDS., com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper: Failed to connect to service endpoint

[Akshaya-T]

Overview

• I am experimenting with Kafka UI.
• I am trying to connect to a AWS MSK cluster in Kafka UI from a self managed kops cluster.
• I can able to access the kafka clusters from the pod using awscli as the pod is annotated with kube2iam role.
• But I am facing the following error from kafka-ui when trying to connect with MSK clusters.

Configuration

aws config file

[default]
role_arn = arn:aws:iam:::role/kafka-ui-kops-kube2iam-role
region = us-east-2
credential_source = Ec2InstanceMetadata
role_session_name = kafka-ui

Helm values file

env:

• name: KAFKA_CLUSTERS_0_NAME
  value: test
• name: KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS
  value: b-1.test.qfyzys.c3.kafka.us-east-2.amazonaws.com:9098
• name: KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL
  value: SASL_SSL
• name: KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM
  value: AWS_MSK_IAM
• name: KAFKA_CLUSTERS_0_PROPERTIES_SASL_CLIENT_CALLBACK_HANDLER_CLASS
  value: software.amazon.msk.auth.iam.IAMClientCallbackHandler
• name: KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG
  value: software.amazon.msk.auth.iam.IAMLoginModule required awsProfileName="default";

---

Error

12-30 16:17:26,130 WARN [kafka-admin-client-thread | kafbat-ui-admin-1735575433-519] s.a.m.a.i.i.MSKCredentialProvider: Exception loading credentials. Retry Attempts: 3
com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [software.amazon.msk.auth.iam.internals.EnhancedProfileCredentialsProvider@66c60036: Failed to load credentials from IMDS., com.amazonaws.auth.AWSCredentialsProviderChain@41a51f30: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName, software.amazon.msk.auth.iam.internals.EnhancedProfileCredentialsProvider@5ee7edcb: Failed to load credentials from IMDS., com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@ea30419: Failed to connect to service endpoint: ]]
at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:136)
at software.amazon.msk.auth.iam.internals.MSKCredentialProvider.loadCredentialsWithRetry(MSKCredentialProvider.java:154)
at software.amazon.msk.auth.iam.internals.MSKCredentialProvider.getCredentials(MSKCredentialProvider.java:141)
at software.amazon.msk.auth.iam.IAMClientCallbackHandler.handleCallback(IAMClientCallbackHandler.java:100)
at software.amazon.msk.auth.iam.IAMClientCallbackHandler.handle(IAMClientCallbackHandler.java:77)
at software.amazon.msk.auth.iam.internals.IAMSaslClient.generateClientMessage(IAMSaslClient.java:139)
at software.amazon.msk.auth.iam.internals.IAMSaslClient.evaluateChallenge(IAMSaslClient.java:96)
at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslToken$1(SaslClientAuthenticator.java:534)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/javax.security.auth.Subject.doAs(Subject.java:439)
at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:534)
at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslClientToken(SaslClientAuthenticator.java:433)
at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendInitialToken(SaslClientAuthenticator.java:332)
at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:273)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:181)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:571)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1413)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1344)
at java.base/java.lang.Thread.run(Thread.java:840)
2024-1