Erratic behaviour with MSK

[DavidConnack]

Hi All

Ive been struggling with the stability of kafka-ui with a MSK cluster.
Cluster is version 2.6.2
Broker type: kafka.t3.small
No brokers: 2

Its a dev cluster with very minimal load.
Often the topics wont load, (says there are no topics), messages dont appear.
A lot of different error messages in the logs:

2022-10-18 05:58:49,148 INFO  [main] c.p.k.u.KafkaUiApplication: Starting KafkaUiApplication using Java 13.0.9 on kafka-ui-696b595b88-wrqwn with PID 1 (/kafka-ui-api.jar started by kafkaui in /)
2022-10-18 05:58:49,151 DEBUG [main] c.p.k.u.KafkaUiApplication: Running with Spring Boot v2.6.3, Spring v5.3.15
2022-10-18 05:58:49,151 INFO  [main] c.p.k.u.KafkaUiApplication: No active profile set, falling back to default profiles: default
2022-10-18 05:58:52,110 INFO  [main] o.s.d.r.c.RepositoryConfigurationDelegate: Bootstrapping Spring Data LDAP repositories in DEFAULT mode.
2022-10-18 05:58:52,218 INFO  [main] o.s.d.r.c.RepositoryConfigurationDelegate: Finished Spring Data repository scanning in 94 ms. Found 0 LDAP repository interfaces.
2022-10-18 05:58:53,528 INFO  [main] c.p.k.u.s.DeserializationService: Using SimpleRecordSerDe for cluster 'mscv3-dev'
2022-10-18 05:58:55,399 INFO  [main] o.s.b.a.e.w.EndpointLinksResolver: Exposing 2 endpoint(s) beneath base path '/actuator'
2022-10-18 05:58:55,640 INFO  [main] o.h.v.i.u.Version: HV000001: Hibernate Validator 6.2.0.Final
2022-10-18 05:58:55,988 INFO  [main] o.s.b.a.s.r.ReactiveUserDetailsServiceAutoConfiguration: 

Using generated security password: f630ed03-0438-4ec3-8378-5c7488fb4ce7

2022-10-18 05:58:56,126 WARN  [main] c.p.k.u.c.a.DisabledAuthSecurityConfig: Authentication is disabled. Access will be unrestricted.
2022-10-18 05:58:56,382 INFO  [main] o.s.l.c.s.AbstractContextSource: Property 'userDn' not set - anonymous context will be used for read-write operations
2022-10-18 05:58:56,872 INFO  [main] o.s.b.w.e.n.NettyWebServer: Netty started on port 8080
2022-10-18 05:58:56,924 INFO  [main] c.p.k.u.KafkaUiApplication: Started KafkaUiApplication in 9.476 seconds (JVM running for 11.965)
2022-10-18 05:58:56,983 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: mscv3-dev
2022-10-18 05:58:57,005 INFO  [parallel-1] o.a.k.c.a.AdminClientConfig: AdminClientConfig values: 
	bootstrap.servers = [b-2.mscv3dev.redacted.redacted.kafka.us-east-1.amazonaws.com:9098, b-1.mscv3dev.redacted.redacted.kafka.us-east-1.amazonaws.com:9098]
	client.dns.lookup = use_all_dns_ips
	client.id = 
	connections.max.idle.ms = 300000
	default.api.timeout.ms = 60000
	metadata.max.age.ms = 300000
	metric.reporters = []
	metrics.num.samples = 2
	metrics.recording.level = INFO
	metrics.sample.window.ms = 30000
	receive.buffer.bytes = 65536
	reconnect.backoff.max.ms = 1000
	reconnect.backoff.ms = 50
	request.timeout.ms = 30000
	retries = 2147483647
	retry.backoff.ms = 100
	sasl.client.callback.handler.class = class software.amazon.msk.auth.iam.IAMClientCallbackHandler
	sasl.jaas.config = [hidden]
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	sasl.kerberos.min.time.before.relogin = 60000
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	sasl.kerberos.ticket.renew.window.factor = 0.8
	sasl.login.callback.handler.class = null
	sasl.login.class = null
	sasl.login.refresh.buffer.seconds = 300
	sasl.login.refresh.min.period.seconds = 60
	sasl.login.refresh.window.factor = 0.8
	sasl.login.refresh.window.jitter = 0.05
	sasl.mechanism = AWS_MSK_IAM
	security.protocol = SASL_SSL
	security.providers = null
	send.buffer.bytes = 131072
	socket.connection.setup.timeout.max.ms = 30000
	socket.connection.setup.timeout.ms = 10000
	ssl.cipher.suites = null
	ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
	ssl.endpoint.identification.algorithm = https
	ssl.engine.factory.class = null
	ssl.key.password = null
	ssl.keymanager.algorithm = SunX509
	ssl.keystore.certificate.chain = null
	ssl.keystore.key = null
	ssl.keystore.location = null
	ssl.keystore.password = null
	ssl.keystore.type = JKS
	ssl.protocol = TLSv1.3
	ssl.provider = null
	ssl.secure.random.implementation = null
	ssl.trustmanager.algorithm = PKIX
	ssl.truststore.certificates = null
	ssl.truststore.location = null
	ssl.truststore.password = null
	ssl.truststore.type = JKS

2022-10-18 05:58:57,485 INFO  [parallel-1] o.a.k.c.s.a.AbstractLogin: Successfully logged in.
2022-10-18 05:58:57,764 INFO  [parallel-1] o.a.k.c.u.AppInfoParser: Kafka version: 2.8.0
2022-10-18 05:58:57,766 INFO  [parallel-1] o.a.k.c.u.AppInfoParser: Kafka commitId: ebb1d6e21cc92130
2022-10-18 05:58:57,767 INFO  [parallel-1] o.a.k.c.u.AppInfoParser: Kafka startTimeMs: 1666072737757
2022-10-18 06:01:06,742 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Metrics updated for cluster: mscv3-dev
2022-10-18 06:01:06,745 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: mscv3-dev
2022-10-18 06:01:35,196 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Metrics updated for cluster: mscv3-dev
2022-10-18 06:01:35,197 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: mscv3-dev
2022-10-18 06:02:26,697 WARN  [parallel-1] o.h.v.i.p.j.JavaBeanExecutable: HV000254: Missing parameter metadata for TopicColumnsToSortDTO(String, int, String), which declares implicit or synthetic parameters. Automatic resolution of generic type information for method parameters may yield incorrect results if multiple parameters have the same erasure. To solve this, compile your code with the '-parameters' flag.
2022-10-18 06:02:26,705 WARN  [parallel-1] o.h.v.i.p.j.JavaBeanExecutable: HV000254: Missing parameter metadata for SortOrderDTO(String, int, String), which declares implicit or synthetic parameters. Automatic resolution of generic type information for method parameters may yield incorrect results if multiple parameters have the same erasure. To solve this, compile your code with the '-parameters' flag.
2022-10-18 06:02:28,178 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Metrics updated for cluster: mscv3-dev
2022-10-18 06:02:28,179 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: mscv3-dev
2022-10-18 06:02:53,024 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Metrics updated for cluster: mscv3-dev
2022-10-18 06:02:53,025 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: mscv3-dev
2022-10-18 06:03:48,890 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Metrics updated for cluster: mscv3-dev
2022-10-18 06:03:48,891 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: mscv3-dev
2022-10-18 06:04:54,669 ERROR [parallel-1] c.p.k.u.s.MetricsService: Failed to collect cluster mscv3-dev info
org.apache.kafka.common.errors.TimeoutException: Call(callName=describeConfigs, deadlineMs=1666073094657, tries=2, nextAllowedTryMs=1666073094762) timed out at 1666073094662 after 2 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: describeConfigs
2022-10-18 06:04:54,669 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Metrics updated for cluster: mscv3-dev
2022-10-18 06:04:54,675 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: mscv3-dev
2022-10-18 06:05:54,707 ERROR [parallel-1] c.p.k.u.s.MetricsService: Failed to collect cluster mscv3-dev info
org.apache.kafka.common.errors.TimeoutException: Call(callName=describeConfigs, deadlineMs=1666073154701, tries=1, nextAllowedTryMs=1666073154804) timed out at 1666073154704 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: describeConfigs
2022-10-18 06:05:54,710 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Metrics updated for cluster: mscv3-dev
2022-10-18 06:05:54,710 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: mscv3-dev
2022-10-18 06:06:50,805 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Metrics updated for cluster: mscv3-dev
2022-10-18 06:06:50,806 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: mscv3-dev
2022-10-18 06:07:49,033 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Metrics updated for cluster: mscv3-dev
2022-10-18 06:07:49,036 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: mscv3-dev
2022-10-18 06:08:12,786 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Metrics updated for cluster: mscv3-dev
2022-10-18 06:08:12,789 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: mscv3-dev
2022-10-18 06:08:33,502 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Metrics updated for cluster: mscv3-dev
2022-10-18 06:08:33,505 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: mscv3-dev
2022-10-18 06:09:33,590 INFO  [kafka-admin-client-thread | adminclient-1] o.a.k.c.a.i.AdminMetadataManager: [AdminClient clientId=adminclient-1] Metadata update failed
org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1666073373588, tries=1, nextAllowedTryMs=1666073373690) timed out at 1666073373590 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata
2022-10-18 06:09:52,950 ERROR [parallel-1] c.p.k.u.s.MetricsService: Failed to collect cluster mscv3-dev info
org.apache.kafka.common.errors.TimeoutException: Call(callName=describeConfigs, deadlineMs=1666073392943, tries=1, nextAllowedTryMs=1666073393049) timed out at 1666073392949 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: describeConfigs
2022-10-18 06:09:52,950 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Metrics updated for cluster: mscv3-dev
2022-10-18 06:09:52,953 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: mscv3-dev
2022-10-18 06:10:27,773 INFO  [kafka-admin-client-thread | adminclient-1] o.a.k.c.a.i.AdminMetadataManager: [AdminClient clientId=adminclient-1] Metadata update failed
org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1666073427768, tries=1, nextAllowedTryMs=1666073427873) timed out at 1666073427773 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata
2022-10-18 06:10:52,954 ERROR [kafka-admin-client-thread | adminclient-1] c.p.k.u.s.MetricsService: Failed to collect cluster mscv3-dev info
org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1666073452953, tries=1, nextAllowedTryMs=1666073453054) timed out at 1666073452954 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: listNodes
2022-10-18 06:10:52,954 DEBUG [kafka-admin-client-thread | adminclient-1] c.p.k.u.s.ClustersMetricsScheduler: Metrics updated for cluster: mscv3-dev
2022-10-18 06:10:52,955 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: mscv3-dev
2022-10-18 06:12:20,973 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Metrics updated for cluster: mscv3-dev
2022-10-18 06:12:20,974 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: mscv3-dev
2022-10-18 06:12:40,559 WARN  [parallel-1] o.h.v.i.p.j.JavaBeanExecutable: HV000254: Missing parameter metadata for SeekTypeDTO(String, int, String), which declares implicit or synthetic parameters. Automatic resolution of generic type information for method parameters may yield incorrect results if multiple parameters have the same erasure. To solve this, compile your code with the '-parameters' flag.
2022-10-18 06:12:40,647 WARN  [parallel-1] o.h.v.i.p.j.JavaBeanExecutable: HV000254: Missing parameter metadata for MessageFilterTypeDTO(String, int, String), which declares implicit or synthetic parameters. Automatic resolution of generic type information for method parameters may yield incorrect results if multiple parameters have the same erasure. To solve this, compile your code with the '-parameters' flag.
2022-10-18 06:12:40,658 WARN  [parallel-1] o.h.v.i.p.j.JavaBeanExecutable: HV000254: Missing parameter metadata for SeekDirectionDTO(String, int, String), which declares implicit or synthetic parameters. Automatic resolution of generic type information for method parameters may yield incorrect results if multiple parameters have the same erasure. To solve this, compile your code with the '-parameters' flag.
2022-10-18 06:12:43,609 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Metrics updated for cluster: mscv3-dev
2022-10-18 06:12:43,614 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: mscv3-dev
2022-10-18 06:12:43,709 INFO  [boundedElastic-3] o.a.k.c.c.ConsumerConfig: ConsumerConfig values: 
	allow.auto.create.topics = false
	auto.commit.interval.ms = 5000
	auto.offset.reset = earliest
	bootstrap.servers = [b-2.redacted.redacted.c19.kafka.us-east-1.amazonaws.com:9098, b-1.redacted.redacted.c19.kafka.us-east-1.amazonaws.com:9098]
	check.crcs = true
	client.dns.lookup = use_all_dns_ips
	client.id = kafka-ui-b92a0467-420a-4a5d-8a67-a16cc3d62bc8
	client.rack = 
	connections.max.idle.ms = 540000
	default.api.timeout.ms = 60000
	enable.auto.commit = false
	exclude.internal.topics = true
	fetch.max.bytes = 52428800
	fetch.max.wait.ms = 500
	fetch.min.bytes = 1
	group.id = null
	group.instance.id = null
	heartbeat.interval.ms = 3000
	interceptor.classes = []
	internal.leave.group.on.close = true
	internal.throw.on.fetch.stable.offset.unsupported = false
	isolation.level = read_uncommitted
	key.deserializer = class org.apache.kafka.common.serialization.BytesDeserializer
	max.partition.fetch.bytes = 1048576
	max.poll.interval.ms = 300000
	max.poll.records = 500
	metadata.max.age.ms = 300000
	metric.reporters = []
	metrics.num.samples = 2
	metrics.recording.level = INFO
	metrics.sample.window.ms = 30000
	partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
	receive.buffer.bytes = 65536
	reconnect.backoff.max.ms = 1000
	reconnect.backoff.ms = 50
	request.timeout.ms = 30000
	retry.backoff.ms = 100
	sasl.client.callback.handler.class = class software.amazon.msk.auth.iam.IAMClientCallbackHandler
	sasl.jaas.config = [hidden]
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	sasl.kerberos.min.time.before.relogin = 60000
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	sasl.kerberos.ticket.renew.window.factor = 0.8
	sasl.login.callback.handler.class = null
	sasl.login.class = null
	sasl.login.refresh.buffer.seconds = 300
	sasl.login.refresh.min.period.seconds = 60
	sasl.login.refresh.window.factor = 0.8
	sasl.login.refresh.window.jitter = 0.05
	sasl.mechanism = AWS_MSK_IAM
	security.protocol = SASL_SSL
	security.providers = null
	send.buffer.bytes = 131072
	session.timeout.ms = 10000
	socket.connection.setup.timeout.max.ms = 30000
	socket.connection.setup.timeout.ms = 10000
	ssl.cipher.suites = null
	ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
	ssl.endpoint.identification.algorithm = https
	ssl.engine.factory.class = null
	ssl.key.password = null
	ssl.keymanager.algorithm = SunX509
	ssl.keystore.certificate.chain = null
	ssl.keystore.key = null
	ssl.keystore.location = null
	ssl.keystore.password = null
	ssl.keystore.type = JKS
	ssl.protocol = TLSv1.3
	ssl.provider = null
	ssl.secure.random.implementation = null
	ssl.trustmanager.algorithm = PKIX
	ssl.truststore.certificates = null
	ssl.truststore.location = null
	ssl.truststore.password = null
	ssl.truststore.type = JKS
	value.deserializer = class org.apache.kafka.common.serialization.BytesDeserializer

2022-10-18 06:12:43,792 INFO  [boundedElastic-3] o.a.k.c.u.AppInfoParser: Kafka version: 2.8.0
2022-10-18 06:12:43,793 INFO  [boundedElastic-3] o.a.k.c.u.AppInfoParser: Kafka commitId: ebb1d6e21cc92130
2022-10-18 06:12:43,793 INFO  [boundedElastic-3] o.a.k.c.u.AppInfoParser: Kafka startTimeMs: 1666073563792
2022-10-18 06:12:43,802 DEBUG [boundedElastic-3] c.p.k.u.e.TailingEmitter: Starting topic tailing
2022-10-18 06:13:14,568 INFO  [boundedElastic-3] o.a.k.c.n.Selector: [Consumer clientId=kafka-ui-b92a0467-420a-4a5d-8a67-a16cc3d62bc8, groupId=null] Failed authentication with b-1.redacted.redacted.c19.kafka.us-east-1.amazonaws.com/172.29.4.149 ([789b74ff-5385-4ee9-a9e6-c3d4e828cd38]: Internal error)
2022-10-18 06:13:14,570 ERROR [boundedElastic-3] o.a.k.c.NetworkClient: [Consumer clientId=kafka-ui-b92a0467-420a-4a5d-8a67-a16cc3d62bc8, groupId=null] Connection to node -2 (b-1.redacted.us-east-1.amazonaws.com/172.29.4.149:9098) failed authentication due to: [789b74ff-5385-4ee9-a9e6-c3d4e828cd38]: Internal error
2022-10-18 06:13:14,570 WARN  [boundedElastic-3] o.a.k.c.NetworkClient: [Consumer clientId=kafka-ui-b92a0467-420a-4a5d-8a67-a16cc3d62bc8, groupId=null] Bootstrap broker b-1.mscv3dev.h5besb.c19.kafka.us-east-1.amazonaws.com:9098 (id: -2 rack: null) disconnected
2022-10-18 06:13:14,582 INFO  [boundedElastic-3] o.a.k.c.m.Metrics: Metrics scheduler closed
2022-10-18 06:13:14,582 INFO  [boundedElastic-3] o.a.k.c.m.Metrics: Closing reporter org.apache.kafka.common.metrics.JmxReporter
2022-10-18 06:13:14,583 INFO  [boundedElastic-3] o.a.k.c.m.Metrics: Metrics reporters closed
2022-10-18 06:13:14,589 INFO  [boundedElastic-3] o.a.k.c.u.AppInfoParser: App info kafka.consumer for kafka-ui-b92a0467-420a-4a5d-8a67-a16cc3d62bc8 unregistered
2022-10-18 06:13:14,589 ERROR [boundedElastic-3] c.p.k.u.e.TailingEmitter: Error consuming ConsumerPosition(seekType=LATEST, seekTo={group-policy-topic-1=8, group-policy-topic-0=4}, seekDirection=TAILING)
org.apache.kafka.common.errors.SaslAuthenticationException: [789b74ff-5385-4ee9-a9e6-c3d4e828cd38]: Internal error
2022-10-18 06:13:14,608 ERROR [boundedElastic-3] o.s.b.a.w.r.e.AbstractErrorWebExceptionHandler: [d488e24a-206]  500 Server Error for HTTP GET "/api/clusters/mscv3-dev/topics/group-policy-topic/messages?filterQueryType=STRING_CONTAINS&attempt=2&limit=100&seekDirection=TAILING&seekType=LATEST&seekTo=0::4,1::8"
org.apache.kafka.common.errors.SaslAuthenticationException: [789b74ff-5385-4ee9-a9e6-c3d4e828cd38]: Internal error
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	*__checkpoint ⇢ Handler com.provectus.kafka.ui.controller.MessagesController#getTopicMessages(String, String, SeekTypeDTO, List, Integer, String, MessageFilterTypeDTO, SeekDirectionDTO, ServerWebExchange) [DispatcherHandler]
	*__checkpoint ⇢ com.provectus.kafka.ui.config.ReadOnlyModeFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ com.provectus.kafka.ui.config.CustomWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ org.springframework.security.web.server.authorization.AuthorizationWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ org.springframework.security.web.server.authorization.ExceptionTranslationWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ org.springframework.security.web.server.authentication.logout.LogoutWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ org.springframework.security.web.server.savedrequest.ServerRequestCacheWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ org.springframework.security.web.server.context.SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ org.springframework.security.web.server.context.ReactorContextWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ org.springframework.security.web.server.header.HttpHeaderWriterWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ org.springframework.security.config.web.server.ServerHttpSecurity$ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain]
	*__checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
	*__checkpoint ⇢ HTTP GET "/kafka/api/clusters/mscv3-dev/topics/group-policy-topic/messages?filterQueryType=STRING_CONTAINS&attempt=2&limit=100&seekDirection=TAILING&seekType=LATEST&seekTo=0::4,1::8" [ExceptionHandlingWebHandler]

Auth is done using IAM_AUTH
I have given the most liberal permissions (for testing):

{
    "Statement": [
        {
            "Action": [
                "kafka-cluster:*"
            ],
            "Effect": "Allow",
            "Resource": [
                "*"
            ]
        },
        {
            "Action": [
                "kafka-cluster:Connect",
                "kafka-cluster:AlterCluster",
                "kafka-cluster:DescribeCluster"
            ],
            "Effect": "Allow",
            "Resource": [
                "*"
            ]
        },
        {
            "Action": [
                "kafka-cluster:*Topic*",
                "kafka-cluster:WriteData",
                "kafka-cluster:ReadData"
            ],
            "Effect": "Allow",
            "Resource": [
                "*"
            ]
        },
        {
            "Action": [
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeGroup"
            ],
            "Effect": "Allow",
            "Resource": [
                "*"
            ]
        }
    ],
    "Version": "2012-10-17"
}

Is there something obvious that I am missing

[Haarolean]

Hi, which version of the app are you running?

[DavidConnack]

vv0.4.0
(
521ba0c
)
Knew i forgot something :)

[Haarolean]

@DavidConnack please try the latest one (master-labeled docker image for example).

[DavidConnack]

I was using latest...
I have switched to master, will see if there is any improvement and update.
Thanks

[DavidConnack]

@Haarolean
I have now run into this issue, as I am utilizing SERVER_SERVLET_CONTEXT_PATH
Is there an eta on a fix for it?
Thank you

[Haarolean]

@DavidConnack yeah we'll get on it soon