[Question] How to use TLS with AWS ALB type ingress? #10968

dicky-yuen · 2024-12-19T12:47:33Z

dicky-yuen
Dec 19, 2024

Hi Teams,

I'm trying to get Kafka external endpoint working with TLS, but it seems like cannot work even I use keytool import AWS ALB ingress CA cert and Kafka cluster CA cert.

Environment:
AWS EKS version 1.27
strimzi-kafka-operator: 0.45.0
kafka version: 3.9.0

Here's what I tried:

Deploy Kafka cluster

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-cluster
  namespace: kafka
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:
  kafka:
    version: 3.9.0
    metadataVersion: 3.9.0
    replicas: 3
    resources:
      requests:
        memory: 2Gi
        cpu: "1"
      limits:
        memory: 2Gi
        cpu: "1"
    logging: 
      type: inline
      loggers:
        rootLogger.level: INFO
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
      - name: external
        port: 9095
        type: ingress
        tls: true
        configuration:
          class: alb
          bootstrap:
            annotations:
              alb.ingress.kubernetes.io/target-type: ip
              alb.ingress.kubernetes.io/subnets: subnet-xxxxxx, subnet-xxxxx
              alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:eu-west-2:xxxxx:certificate/xxxxx
            host: bootstrap.test.tech
          brokers:
          - broker: 0
            annotations:
              alb.ingress.kubernetes.io/target-type: ip
              alb.ingress.kubernetes.io/subnets: subnet-xxxxxx, subnet-xxxxx
              alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:eu-west-2:xxxxx:certificate/xxxxx
            host: broker-0.test.tech
          - broker: 1
            annotations:
              alb.ingress.kubernetes.io/target-type: ip
              alb.ingress.kubernetes.io/subnets: subnet-xxxxxx, subnet-xxxxx
              alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:eu-west-2:xxxxx:certificate/xxxxx
            host: broker-1.test.tech
          - broker: 2
            annotations:
              alb.ingress.kubernetes.io/target-type: ip
              alb.ingress.kubernetes.io/subnets: subnet-xxxxxx, subnet-xxxxx
              alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:eu-west-2:xxxxx:certificate/xxxxx
            host: broker-2.test.tech
    jvmOptions:
      -Xms: 1024m
      -Xmx: 1024m
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      compression.type: "lz4"
      default.replication.factor: 2
      num.partitions: 16
      min.insync.replicas: 2
      log.retention.ms: 30000
      log.roll.hours: 1
      log.retention.check.interval.ms: 300000
  entityOperator:
    topicOperator:
      watchedNamespace: kafka
      reconciliationIntervalMs: 60000
      resources:
        requests:
          memory: 256Mi
          cpu: "0.5"
        limits:
          memory: 256Mi
          cpu: "0.5"
      logging:
        type: inline
        loggers:
          rootLogger.level: INFO
    userOperator:
      watchedNamespace: kafka
      reconciliationIntervalMs: 60000
      resources:
        requests:
          memory: 256Mi
          cpu: "0.5"
        limits:
          memory: 256Mi
          cpu: "0.5"
      logging:
        type: inline
        loggers:
          rootLogger.level: INFO

reproduce steps:

kubectl get secret kafka-cluster-cluster-ca-cert -o jsonpath='{.data.ca.crt}' | base64 -d > ca.crt
keytool -import -trustcacerts -alias root -file ca.crt -keystore truststore.jks -storepass password -noprompt
bin/kafka-console-producer.sh --broker-list bootstrap.test.tech:443 --producer-property security.protocol=SSL --producer-property ssl.truststore.password=password --producer-property ssl.truststore.location=./truststore.jks --topic test-topic

Script error:
[2024-12-19 16:56:16,560] ERROR [Producer clientId=console-producer] Connection to node -1 (bootstrap.test.tech/xxxx:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient) [2024-12-19 16:56:16,561] WARN [Producer clientId=console-producer] Bootstrap broker bootstrap.test.tech:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

Questions:
I tested internal TLS on port 9093, and it functioned as expected. I suspect the issue lies with the AWS ALB. Even after importing the AWS ALB CA certificate, the problem persists.

Are there any potential solutions to enable AWS ALB ingress type to work with TLS?
Can Kafka Bridge perfectly replace the Kafka script?

Thanks.

Answered by scholzj

Dec 20, 2024

It is not clear how is your client configured. I also doubt you should use service.beta.kubernetes.io/aws-load-balancer-ssl-cert? What do you expect that to do?

View full answer

scholzj · 2024-12-19T13:12:30Z

scholzj
Dec 19, 2024
Maintainer

I don't think you can use it. As far as I know, Amazon ALB is a layer 7 load balancer for HTTP traffic. It cannot handle TCP traffic or TLS passthrough traffic. You need to use classic load balancers or NLBs.

5 replies

dicky-yuen Dec 20, 2024
Author

Hi @scholzj
Thanks for your update.
I switched AWS ALB to NLB, and reproduced the steps which it got disconnected.

This is my updated YAML file.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-cluster
  namespace: kafka
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:
  kafka:
    version: 3.9.0
    metadataVersion: 3.9.0
    replicas: 3
    resources:
      requests:
        memory: 2Gi
        cpu: "1"
      limits:
        memory: 2Gi
        cpu: "1"
    logging: 
      type: inline
      loggers:
        rootLogger.level: INFO
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
      - name: external
        port: 9094
        type: loadbalancer
        tls: true
        configuration:
          bootstrap:
            annotations:
              service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:eu-west-2:xxx:certificate/xxx"
              service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
              service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "9094"
              service.beta.kubernetes.io/aws-load-balancer-subnets: "subnet-xxx, subnet-xxx"
            alternativeNames:
              - bootstrap.staging.tech
          brokers:
          - broker: 0
            annotations:
              service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:eu-west-2:xxx:certificate/xxx"
              service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
              service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "9094"
              service.beta.kubernetes.io/aws-load-balancer-subnets: "subnet-xxx, subnet-xxx"
            advertisedHost: broker-0.staging.tech
          - broker: 1
            annotations:
              service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:eu-west-2:xxx:certificate/xxx"
              service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
              service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "9094"
              service.beta.kubernetes.io/aws-load-balancer-subnets: "subnet-xxx, subnet-xxx"
            advertisedHost: broker-1.staging.tech
          - broker: 2
            annotations:
              service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:eu-west-2:xxx:certificate/xxx"
              service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
              service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "9094"
              service.beta.kubernetes.io/aws-load-balancer-subnets: "subnet-xxx, subnet-xxx"
            advertisedHost: broker-2.staging.tech
    jvmOptions:
      -Xms: 1024m
      -Xmx: 1024m
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      compression.type: "lz4"
      default.replication.factor: 2
      num.partitions: 16
      min.insync.replicas: 2
      log.retention.ms: 30000
      log.roll.hours: 1
      log.retention.check.interval.ms: 300000
  entityOperator:
    topicOperator:
      watchedNamespace: kafka
      reconciliationIntervalMs: 60000
      resources:
        requests:
          memory: 256Mi
          cpu: "0.5"
        limits:
          memory: 256Mi
          cpu: "0.5"
      logging:
        type: inline
        loggers:
          rootLogger.level: INFO
    userOperator:
      watchedNamespace: kafka
      reconciliationIntervalMs: 60000
      resources:
        requests:
          memory: 256Mi
          cpu: "0.5"
        limits:
          memory: 256Mi
          cpu: "0.5"
      logging:
        type: inline
        loggers:
          rootLogger.level: INFO

reproduce steps:

kubectl get secret kafka-cluster-cluster-ca-cert -o jsonpath='{.data.ca.crt}' | base64 -d > ca.crt
keytool -import -trustcacerts -alias root -file ca.crt -keystore truststore.jks -storepass password -noprompt
../kafka_2.13-3.5.0/bin/kafka-console-producer.sh --bootstrap-server bootstrap.staging.tech:9094 --topic test-topic --producer.config client.properties

Script error:
[2024-12-19 16:56:16,560] ERROR [Producer clientId=console-producer] Connection to node -1 (bootstrap.staging.tech/xxxx:9094) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient) [2024-12-19 16:56:16,561] WARN [Producer clientId=console-producer] Bootstrap broker bootstrap.staging.tech:9094 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

I still got SSL handshake failed, and I tested tls: false works normally. Do I need to store the Kafka cluster CA certificate to AWS cert store?

Did I configure anything wrong?

scholzj Dec 20, 2024
Maintainer

It is not clear how is your client configured. I also doubt you should use service.beta.kubernetes.io/aws-load-balancer-ssl-cert? What do you expect that to do?

Answer selected by dicky-yuen

dicky-yuen Dec 20, 2024
Author

You are right. I shouldn't use service.beta.kubernetes.io/aws-load-balancer-ssl-cert. I need to use the Kafka cluster CA to communicate with the broker. After removing service.beta.kubernetes.io/aws-load-balancer-ssl-cert and service.beta.kubernetes.io/aws-load-balancer-ssl-ports, I can now use TLS.

Thank you so much.

dicky-yuen Dec 20, 2024
Author

I have one more question. If I want to use my self-signed CA instead of Kafka cluster generated CA.
How can I achieve this?
is it put my certificate under brokerCertChainAndKey?

listeners:
  #...
  - name: external3
    port: 9094
    type: loadbalancer
    tls: true
    configuration:
      brokerCertChainAndKey:
        secretName: my-secret
        certificate: my-listener-certificate.crt
        key: my-listener-key.key
# ...

Thanks.

scholzj Dec 20, 2024
Maintainer

Yes, you can specify it like this.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

[Question] How to use TLS with AWS ALB type ingress? #10968

{{title}}

Replies: 1 comment 5 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Strimzi

[Question] How to use TLS with AWS ALB type ingress? #10968

dicky-yuen Dec 19, 2024

Replies: 1 comment · 5 replies

scholzj Dec 19, 2024 Maintainer

dicky-yuen Dec 20, 2024 Author

scholzj Dec 20, 2024 Maintainer

dicky-yuen Dec 20, 2024 Author

dicky-yuen Dec 20, 2024 Author

scholzj Dec 20, 2024 Maintainer

dicky-yuen
Dec 19, 2024

Replies: 1 comment 5 replies

scholzj
Dec 19, 2024
Maintainer

dicky-yuen Dec 20, 2024
Author

scholzj Dec 20, 2024
Maintainer

dicky-yuen Dec 20, 2024
Author

dicky-yuen Dec 20, 2024
Author

scholzj Dec 20, 2024
Maintainer