Hbase LoadBalancer ports are mapped wrong

[Maleware]

Affected Stackable version

25.7.0

Affected Apache HBase version

2.6.2-stackable25.7.0

Current and expected behavior

Setup

We have a cluster on Ionos ( reproduceable on kind using cloud-provider-kind ) with public exposition of nodes enabled. We want to expose hbase-master's ui-https or ui-http endpoints to the world using a LoadBalancer wrapped into a custom listenerClass:

---
apiVersion: listeners.stackable.tech/v1alpha1
kind: ListenerClass
metadata:
  name: lb-stable
spec:
  serviceType: LoadBalancer

we get a service as expected:

stackable-products  hbase-master-default-0-listener          LoadBalancer  100.64.20.247  217.154.164.205  master:16000►30091 ui-http:16010►32301

using this manifest (pre-requisites are met e.g. HDFS and ZK ):

---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperZnode
metadata:
  name: hbase-znode
  namespace: stackable-products
spec:
  clusterRef:
    name: zookeeper
---
apiVersion: hbase.stackable.tech/v1alpha1
kind: HbaseCluster
metadata:
  name: hbase
  namespace: stackable-products
spec:
  image:
    productVersion: 2.6.2
  clusterConfig:
    hdfsConfigMapName: simple-hdfs
    zookeeperConfigMapName: hbase-znode
  masters:
    config:
      listenerClass: lb-stable  # <-- ListenerClass used for master
      logging:
        enableVectorAgent: true
        containers:
          hbase:
            console:
              level: NONE
          vector:
            console:
              level: NONE
    roleGroups:
      default:
        replicas: 1
  regionServers:
    config:
      logging:
        enableVectorAgent: true
        containers:
          hbase:
            console:
              level: NONE
          vector:
            console:
              level: NONE
    roleGroups:
      default:
        replicas: 2
    configOverrides:
      hbase-site.xml:
        hbase.coprocessor.region.classes: tech.stackable.hbase.OpenPolicyAgentAccessController,org.apache.hadoop.hbase.security.token.TokenProvider

However, Hbase remains unavailable. I will only outline the http-ui example, same is true if Tls is enabled.

Current situation

Shelling into the Hbase masters pod reveals the following:

stackable@hbase-master-default-0 /stackable/hbase-2.6.2-stackable25.7.0 $ cat ../listener/default-address/address
217.154.164.189

and

 cat ../listener/default-address/ports/ui-http
16010

where the latter is the wrong port from the LoadBalancer.

Expected Situation

I'd expect the LB port 32301 being present in the pod.

Possible solution

No response

Additional context

Kafka would do it correct finding this situation ( with the same listenerClass )

stackable-products  kafka-broker-default-bootstrap           LoadBalancer  100.64.33.254   217.154.164.190   bootstrap:9095►31102 kafka-tls:9093►31541 metrics:9606►30484

Checking the kafka pod reveals:

stackable@kafka-broker-default-0 /stackable/kafka_2.13-3.9.0-stackable25.7.0 $ cat ../listener-bootstrap/default-address/ports/bootstrap
9095

Environment

No response

Would you like to work on fixing this bug?

None

[sbernauer]

Can you please paste the full YAML of the listener (including status) if possible?

[Maleware]

Sure:

apiVersion: listeners.stackable.tech/v1alpha1
kind: Listener
metadata:
  creationTimestamp: "2025-10-07T16:37:57Z"
  generation: 1
  labels:
    app.kubernetes.io/component: master
    app.kubernetes.io/instance: hbase
    app.kubernetes.io/managed-by: hbase.stackable.com_hbasecluster
    app.kubernetes.io/name: hbase
    app.kubernetes.io/role-group: default
    app.kubernetes.io/version: 2.6.2-stackable25.7.0
    stackable.tech/vendor: Stackable
  name: hbase-master-default-0-listener
  namespace: stackable-products
  ownerReferences:
  - apiVersion: v1
    kind: PersistentVolume
    name: pvc-77bdd077-4b56-4265-94f6-e4d779dc5641
    uid: 01ec8b96-3a3b-46bc-b89a-f069fe7f7721
  resourceVersion: "50728608335"
  uid: eb1e09f1-ae73-4b69-933a-a3b49597b55b
spec:
  className: lb-stable
  extraPodSelectorLabels: {}
  ports:
  - name: master
    port: 16000
    protocol: TCP
  - name: ui-http
    port: 16010
    protocol: TCP
  publishNotReadyAddresses: true
status:
  ingressAddresses:
  - address: 217.154.164.205
    addressType: IP
    ports:
      master: 16000
      ui-http: 16010
  nodePorts: null
  serviceName: hbase-master-default-0-listener

[sbernauer]

I'm likely missing something but I'm failing to see the problem 🙈
If you have a LoadBalancer
LoadBalancer 100.64.33.254 217.154.164.190 bootstrap:9095►31102 kafka-tls:9093►31541 metrics:9606►30484
you normally access it using port 9095, 9093 or 9606, which are the advertised enpoints in the Listener status.
This is e.g. how it works on IONOS, GKE, Rhoencloud and others.

Maybe this is a kind thing, which somehow tries to do a LoadBalancer via a NodePort or whatnot?
Does the LoadBalancer work on IONOS?

[Maleware]

It doesn't work with IONOS.

If you look at the LB of Hbase you find the ports being mapped master:16000►30091 ui-http:16010►32301 which then the LB port 32301 should be reflected under default-adresses/ports/ui-http. However it still uses 16010 which is the original port on the pod not the port on the LB.

You picked kafka as example, which does it appropriate.

[maltesander]

you normally access it using port 9095, 9093 or 9606, which are the advertised enpoints in the Listener status. This is e.g. how it works on IONOS, GKE, Rhoencloud and others.

I think its about

status:
  ingressAddresses:
  - address: 217.154.164.205
    addressType: IP
    ports:
      master: 16000
      ui-http: 16010
  nodePorts: null

Shouldnt the status reflect the load balancer port?

[NickLarsenNZ]

@maltesander, I just looked into this with @Maleware and @sbernauer...

Shouldnt the status reflect the load balancer port?

16010 is the Service (therefore LB) port, not the NodePort.

@sbernauer made a change to the ListenerClass at the same time, which I believe made it accessible.

[Maleware]

It was actually externalTrafficPolicy: Cluster set to local which is an implementation detail of IONOS. Thus it wasn't a bug, it was IONOS...

[sbernauer]

IONOS requires us to set ListenerClass.spec.serviceExternalTrafficPolicy to Cluster: stackabletech/listener-operator#196
Once we did that everything worked fine
@Maleware has a TODO to

1. Add a troubleshooting to https://docs.stackable.tech/home/stable/listener-operator/ (similar to https://docs.stackable.tech/home/stable/nifi/troubleshooting/)
2. Add it to https://docs.stackable.tech/home/stable/kubernetes/ionos-managed-k8s/ as well

[maltesander]

16010 is the Service (therefore LB) port, not the NodePort.

Nice its solved! @NickLarsenNZ i still dont get this. This is the internal port hbase exposes. NOT the LB port?
You basically have a combination of external IP with internal port...

Edit: No matter what we set for ionos, should this even be happening implementation wise?

[NickLarsenNZ]

This is the internal port hbase exposes. NOT the LB port?

It is the Service port, right (whether or not that is the pod port)?

I don't have access to the cluster, so maybe @Maleware can answer.

[sbernauer]

Not an issue, it was actually IONOS not supporting externalTrafficPolicy=local. We will likely change that default