Kafka Cruise Control is failing in Rosa HCP

[rajarshp]

Hi Team,

we are trying to install Kafka using RedHat Stream operator, but somehow Cruise Control is failing. It is showing below error but we are not having any clue why it is not able to use this port.

The same config file we applied in Rosa Classic and everything is working as expected.

So could you please help me to fix this issue ?

Kafka Version: 3.7
RedHat Stream: 2.8.0

Error:

Liveness probe failed: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (7) Failed to connect to localhost port 9090: Connection refused

Operator log:

2024-12-30 16:03:26 ERROR VertxUtil:132 - Reconciliation #1361(timer) Kafka(XXX-kafka): Exceeded timeout of 300000ms while waiting for Deployment resource XXXX-kafka-cruise-control in namespace XXXXX to be ready
2024-12-30 16:03:26 ERROR AbstractOperator:283 - Reconciliation #1361(timer) Kafka(XXX-kafka): createOrUpdate failed
io.strimzi.operator.common.TimeoutException: Exceeded timeout of 300000ms while waiting for Deployment resource XXXX-kafka-cruise-control in namespace XXXX-kafka-01 to be ready
  at io.strimzi.operator.cluster.operator.VertxUtil$1.lambda$handle$1(VertxUtil.java:133) ~[io.strimzi.cluster-operator-0.43.0.redhat-00005.jar:0.43.0.redhat-00005]
  at io.vertx.core.impl.future.FutureImpl$4.onFailure(FutureImpl.java:188) ~[io.vertx.vertx-core-4.5.9.redhat-00001.jar:4.5.9.redhat-00001]
  at io.vertx.core.impl.future.FutureBase.lambda$emitFailure$1(FutureBase.java:75) ~[io.vertx.vertx-core-4.5.9.redhat-00001.jar:4.5.9.redhat-00001]
  at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[io.netty.netty-common-4.1.111.Final-redhat-00002.jar:4.1.111.Final-redhat-00002]
  at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[io.netty.netty-common-4.1.111.Final-redhat-00002.jar:4.1.111.Final-redhat-00002]
  at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) ~[io.netty.netty-common-4.1.111.Final-redhat-00002.jar:4.1.111.Final-redhat-00002]
  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) ~[io.netty.netty-transport-4.1.111.Final-redhat-00002.jar:4.1.111.Final-redhat-00002]
  at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994) ~[io.netty.netty-common-4.1.111.Final-redhat-00002.jar:4.1.111.Final-redhat-00002]
  at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.1.111.Final-redhat-00002.jar:4.1.111.Final-redhat-00002]
  at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.111.Final-redhat-00002.jar:4.1.111.Final-redhat-00002]
  at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-12-30 16:03:26 WARN AbstractOperator:556 - Reconciliation #1361(timer) Kafka(test-kafka): Failed to reconcile
io.strimzi.operator.common.TimeoutException: Exceeded timeout of 300000ms while waiting for Deployment resource XXXX-kafka-cruise-control in namespace XXXX-kafka-01 to be ready
  at io.strimzi.operator.cluster.operator.VertxUtil$1.lambda$handle$1(VertxUtil.java:133) ~[io.strimzi.cluster-operator-0.43.0.redhat-00005.jar:0.43.0.redhat-00005]
  at io.vertx.core.impl.future.FutureImpl$4.onFailure(FutureImpl.java:188) ~[io.vertx.vertx-core-4.5.9.redhat-00001.jar:4.5.9.redhat-00001]
  at io.vertx.core.impl.future.FutureBase.lambda$emitFailure$1(FutureBase.java:75) ~[io.vertx.vertx-core-4.5.9.redhat-00001.jar:4.5.9.redhat-00001]
  at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[io.netty.netty-common-4.1.111.Final-redhat-00002.jar:4.1.111.Final-redhat-00002]
  
Config:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: XXXX-kafka
namespace: XXXX-kafka-01
spec:
cruiseControl: {}
kafka:
resources:
requests:
memory: 1Gi
cpu: 500m
limits:
memory: 1Gi
cpu: 500m
version: 3.7.0
replicas: 3
listeners:
- name: plain
port: 9092
type: internal
tls: false
authentication:
type: scram-sha-512
- name: tls
port: 9093
type: internal
tls: true
authentication:
type: scram-sha-512
- name: external
port: 9094
type: route
tls: true
authentication:
type: scram-sha-512
configuration:
bootstrap:
host: kafka-bootstrap.XXXX.com
brokers:
- broker: 0
host: kafka-broker-0.XXXX.com
- broker: 1
host: kafka-broker-1.XXXX.com
- broker: 2
host: kafka-broker-2.XXXX.com
authorization:
type: simple
config:
allow.everyone.if.no.acl.found: false
offsets.topic.replication.factor: 1
transaction.state.log.replication.factor: 1
transaction.state.log.min.isr: 1
default.replication.factor: 1
min.insync.replicas: 1
inter.broker.protocol.version: "3.7"
auto.create.topics.enable: true
message.max.bytes: 150000000
storage:
type: persistent-claim
size: 100Gi
deleteClaim: false
metricsConfig:
type: jmxPrometheusExporter
valueFrom:
configMapKeyRef:
name: kafka-metrics
key: kafka-metrics-config.yml
zookeeper:
replicas: 3
storage:
type: persistent-claim
size: 5Gi
deleteClaim: false
entityOperator:
topicOperator: {}
userOperator: {}

[scholzj]

Without sharing the full logs and configuration, it is hard to help. Also, for Red Hat products, you should really contact Red Hat support in the first place.

[rajarshp]

@scholzj sorry I cannot share the full log. but below is the configuration
We are checking this as well.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: XXXX-kafka
  namespace: XXXX-kafka-01
spec:
  cruiseControl: {}
  kafka:
    resources:
      requests:
        memory: 1Gi
        cpu: 500m
      limits:
        memory: 1Gi
        cpu: 500m
    version: 3.7.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
        authentication:
          type: scram-sha-512
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: scram-sha-512
      - name: external
        port: 9094
        type: route
        tls: true
        authentication:
          type: scram-sha-512
        configuration:
          bootstrap:
            host: kafka-bootstrap.XXXX.com
          brokers:
          - broker: 0
            host: kafka-broker-0.XXXX.com
          - broker: 1
            host: kafka-broker-1.XXXX.com
          - broker: 2
            host: kafka-broker-2.XXXX.com
    authorization:
      type: simple 
    config:
      allow.everyone.if.no.acl.found: false
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
      inter.broker.protocol.version: "3.7"
      auto.create.topics.enable: true
      message.max.bytes: 150000000
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: false
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 5Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}