Kafbat UI does not show MSK cluster info

[hongbo-miao]

I am using MSK 3.7.x KRaft mode. Also, I am also having some issues seeing the cluster in the Kafbat UI.

I can confirm same IAM role permission (granted all readonly permissions for now) works well for AKHQ and Redpanda Console.

Here is my config

yamlApplicationConfig:
  kafka:
    clusters:
      - name: yaml
        bootstrapServers: b-1.xxx.xx.c11.kafka.us-west-2.amazonaws.com:9098,b-2.xxx.xx.c11.kafka.us-west-2.amazonaws.com:9098,b-3.xxx.xx.c11.kafka.us-west-2.amazonaws.com:9098
        properties:
          security.protocol: SASL_SSL
          sasl.mechanism: AWS_MSK_IAM
          sasl.jaas.config: software.amazon.msk.auth.iam.IAMLoginModule required;
          sasl.client.callback.handler.class: software.amazon.msk.auth.iam.IAMClientCallbackHandler
  auth:
    type: disabled
  management:
    health:
      ldap:
        enabled: false
serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxx4xxx:role/KafbatUIRole-hm-kafbat-ui

https://hm-kafbat-ui.example.com/api/authorization returns

{
"rbacEnabled": false,
"userInfo": null
}

https://hm-kafbat-ui.example.com/api/clusters returns

[
  {
    "name": "yaml",
    "defaultCluster": null,
    "status": "offline",
    "lastError": {
      "message": "Error while creating AdminClient for Cluster yaml",
      "stackTrace": "java.lang.IllegalStateException: Error while creating AdminClient for Cluster yaml\n\tat io.kafbat.ui.service.AdminClientServiceImpl.lambda$createAdminClient$5(AdminClientServiceImpl.java:56)\n\tat reactor.core.publisher.Mono.lambda$onErrorMap$28(Mono.java:3783)\n\tat reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)\n\tat reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258)\n\tat reactor.core.publisher.FluxMap$MapConditionalSubscriber.onError(FluxMap.java:265)\n\tat reactor.core.publisher.Operators$MonoSubscriber.onError(Operators.java:1886)\n\tat reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.signalCached(MonoCacheTime.java:340)\n\tat reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.onError(MonoCacheTime.java:363)\n\tat reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:180)\n\tat reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onError(MonoIgnoreThen.java:278)\n\tat reactor.core.publisher.MonoPublishOn$PublishOnSubscriber.run(MonoPublishOn.java:187)\n\tat reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)\n\tat reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:840)\nCaused by: org.apache.kafka.common.errors.SaslAuthenticationException: [4d37b931-0d78-44a1-bf2c-8ebbf55da91a]: Access denied\n"
    },
    "brokerCount": 0,
    "onlinePartitionCount": 0,
    "topicCount": 0,
    "bytesInPerSec": null,
    "bytesOutPerSec": null,
    "readOnly": false,
    "version": "Unknown",
    "features": []
  }
]

[Haarolean]

Can you verify you have all the permissions required assigned to the policy? Docs reference here

[hongbo-miao]

Thanks @Haarolean I confirmed I have all read-only access granted at here:

resource "aws_iam_role_policy" "hm_kafbat_ui_iam_role_policy" {
  name = "${local.aws_iam_role_name_prefix}Policy-${var.kafbat_ui_service_account_name}"
  role = aws_iam_role.hm_kafbat_ui_iam_role.name
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "kafka-cluster:Connect",
          "kafka-cluster:DescribeCluster",
          "kafka-cluster:DescribeClusterDynamicConfiguration",
          "kafka-cluster:DescribeGroup",
          "kafka-cluster:DescribeTopic",
          "kafka-cluster:DescribeTopicDynamicConfiguration",
          "kafka-cluster:DescribeTransactionalId",
          "kafka-cluster:ReadData"
        ]
        Resource = [
          "*"
        ]
      }
    ]
  })
}

[Haarolean]

Sorry, it seems the issue is with authentication, rather than authorization. Make sure your auth details are correct: https://ui.docs.kafbat.io/configuration/authentication/aws-iam

[hongbo-miao]

Hi @Haarolean Here is my config.

I redacted the broker URLs in bootstrapServers, in real, I have them correctly set up.

I noticed in the doc you provided it is sasl.jaas.config: software.amazon.msk.auth.iam.IAMLoginModule required awsProfileName="<PROFILE_NAME>";.

I am using sasl.jaas.config: software.amazon.msk.auth.iam.IAMLoginModule required;
Because I provided service role eks.amazonaws.com/role-arn: arn:aws:iam::272394222652:role/KafbatUIRole-hm-kafbat-ui.

Is it okay?

[githubeto]

Hi @hongbo-miao

I believe you have already resolved this issue, but I will leave a comment just in case.
I have configured it as follows and am able to connect to the cluster using only IAM authentication correctly.

serviceAccount: 
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxx:role/xxxxxxxxxx
  name: "kafka-ui-sa"
envs:
  config:
    KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: b-1.xxxxxxxxxx.xxxxxxxxxx.c2.kafka.ap-northeast-1.amazonaws.com:9098,b-2.xxxxxxxxxx.xxxxxxxxxx.c2.kafka.ap-northeast-1.amazonaws.com:9098,b-3.xxxxxxxxxx.xxxxxxxxxx.c2.kafka.ap-northeast-1.amazonaws.com:9098
    KAFKA_CLUSTERS_0_NAME: XXXXX
    KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: SASL_SSL
    KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM: AWS_MSK_IAM
    KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG: 'software.amazon.msk.auth.iam.IAMLoginModule required;'
    KAFKA_CLUSTERS_0_PROPERTIES_SASL_CLIENT_CALLBACK_HANDLER_CLASS: 'software.amazon.msk.auth.iam.IAMClientCallbackHandler'

If you are using IRSA, this description should be fine.

KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG: 'software.amazon.msk.auth.iam.IAMLoginModule required;'

Originally, Kafbat-UI suggests using the IAM role of the EC2 instance, so I imagine that specifying a profile is necessary.
However, I believe that using IRSA is the best option in terms of security.