1 node(s) didn't have free ports for the requested pod ports

[jeusdi]

Currently, I'm getting this message when I'm trying to deploy bitnami/mongodb helm chart.

Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  23m                  default-scheduler  0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 node(s) didn't match Pod's node affinity.
  Warning  FailedScheduling  2m2s (x20 over 23m)  default-scheduler  0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 node(s) didn't match Pod's node affinity.

I've deployed bitnami/mongodb helm chart using options:

architecture: replicaset
replicaCount: 2
serviceAccount:
  create: true
rbac:
  create: true
externalAccess:
  enabled: true
  service:
    type: LoadBalancer
  autoDiscovery:
    enabled: true
image:
  debug: true

Important helm values are:

externalAccess:
  enabled: true
  service:
    type: LoadBalancer

Just above options do deploy an LoadBalancer for each mongodb node.

In my case, I've deployed two nodes, so I have two LoadBalancers.

$ kubectl get service output:

NAME                            TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)           AGE
mongodb-1-external              LoadBalancer   10.43.129.193   172.19.0.3    27017:30018/TCP   20m
mongodb-0-external              LoadBalancer   10.43.95.227    172.19.0.2    27017:30017/TCP   20m

$ kubectl get service -o=yaml mongodb-0-external output:

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: mongodb
    meta.helm.sh/release-namespace: salut
  creationTimestamp: "2022-01-11T13:59:22Z"
  labels:
    app.kubernetes.io/component: mongodb
    app.kubernetes.io/instance: mongodb
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: mongodb
    helm.sh/chart: mongodb-10.31.2
    pod: mongodb-0
  name: mongodb-0-external
  namespace: salut
  resourceVersion: "65644"
  uid: 94d755a2-d755-4ae5-b797-5474fb38d36c
spec:
  clusterIP: 10.43.95.227
  clusterIPs:
  - 10.43.95.227
  externalTrafficPolicy: Cluster
  ports:
  - name: mongodb
    nodePort: 30017
    port: 27017
    protocol: TCP
    targetPort: mongodb
  publishNotReadyAddresses: true
  selector:
    app.kubernetes.io/component: mongodb
    app.kubernetes.io/instance: mongodb
    app.kubernetes.io/name: mongodb
    statefulset.kubernetes.io/pod-name: mongodb-0
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 172.19.0.2

$ kubectl get service -o=yaml mongodb-1-external output:

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: mongodb
    meta.helm.sh/release-namespace: salut
  creationTimestamp: "2022-01-11T13:59:22Z"
  labels:
    app.kubernetes.io/component: mongodb
    app.kubernetes.io/instance: mongodb
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: mongodb
    helm.sh/chart: mongodb-10.31.2
    pod: mongodb-1
  name: mongodb-1-external
  namespace: salut
  resourceVersion: "65590"
  uid: e345288e-ed9f-4356-8167-d95c0a5bfa0c
spec:
  clusterIP: 10.43.129.193
  clusterIPs:
  - 10.43.129.193
  externalTrafficPolicy: Cluster
  ports:
  - name: mongodb
    nodePort: 30018
    port: 27017
    protocol: TCP
    targetPort: mongodb
  publishNotReadyAddresses: true
  selector:
    app.kubernetes.io/component: mongodb
    app.kubernetes.io/instance: mongodb
    app.kubernetes.io/name: mongodb
    statefulset.kubernetes.io/pod-name: mongodb-1
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 172.19.0.3

My k3d cluster has 2 agents and 1 server:

$ kubectl get nodes output:

NAME                     STATUS   ROLES                  AGE   VERSION
k3d-terraform-agent-1    Ready    <none>                 39h   v1.20.0+k3s2
k3d-terraform-agent-0    Ready    <none>                 39h   v1.20.0+k3s2
k3d-terraform-server-0   Ready    control-plane,master   39h   v1.20.0+k3s2

So, several svclb-mongodb*-like pods are created:

NAME                                             READY   STATUS             RESTARTS   AGE
svclb-mongodb-1-external-98x48                   0/1     Pending            0          27m
svclb-mongodb-1-external-q6fvg                   0/1     Pending            0          27m
svclb-mongodb-0-external-4msdx                   0/1     Pending            0          27m
svclb-mongodb-0-external-dfmp9                   1/1     Running            0          27m
svclb-mongodb-1-external-snfqf                   1/1     Running            0          27m
svclb-mongodb-0-external-g7nml                   1/1     Running            0          27m

Problem here is that some of them are is Pending status.

I've took a look on those Pending pods events:

Events:
  Type     Reason            Age                   From               Message
  ----     ------            ----                  ----               -------
  Warning  FailedScheduling  30m                   default-scheduler  0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 node(s) didn't match Pod's node affinity.
  Warning  FailedScheduling  9m35s (x20 over 30m)  default-scheduler  0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 node(s) didn't match Pod's node affinity.

I don't quite figure out what's exactly hapenning.

• I guess problem arises from both LoadBalancers are requesting for getting 27017 port? Could it be that the problem? How could I solve it?

[iwilltry42]

Hi @jeusdi , thanks for starting this discussion!
You already got it right there. The issue is that both LoadBalancer type services want to use the same port 27017.
The way LoadBalancers are working in K3s is is via klibber-lb which deploys a DaemonSet per such service, blocking the desired port on every node in the cluster (see docs: https://rancher.com/docs/k3s/latest/en/networking/#service-load-balancer).
Workarounds:

• if you really need it to be LoadBalancer type services, then you need to find a way to change one of them to a different port (either somewhere in the Helm chart or after deployment in-cluster via kubectl edit or kubectl patch)
• switch to type: ClusterIP or type: NodePort in your values file

[jeusdi]

While I was trying to troubleshoot the problem, I have tried to use metallb. It seems that it's able to get an "external IP"...

I've deployed k3d using this config file, disabling load balancing and servicelb:

apiVersion: k3d.io/v1alpha3
kind: Simple
name: salut
servers: 1
agents: 2
ports:
  - port: 8000:32080
    nodeFilters:
      - server:0:direct
  - port: 8443:32443
    nodeFilters:
      - server:0:direct
  - port: 9000:32090
    nodeFilters:
      - server:0:direct
  - port: 20017:30017
    nodeFilters:
      - server:0:direct
  - port: 20018:30018
    nodeFilters:
      - server:0:direct
registries:
  config: |
    mirrors:
      "registry.localhost:5000":
        endpoint:
          - http://host.k3d.internal:5000
options:
  k3d:
    wait: true
    timeout: "60s"
    disableLoadbalancer: true
    disableImageVolume: false
    disableRollback: true
  k3s:
    extraArgs:
      - arg: '--disable=traefik,servicelb'
        nodeFilters:
          - server:*
  kubeconfig:
    updateDefaultKubeconfig: true
    switchCurrentContext: true

As you can see:

k3d:
    disableLoadbalancer: true
k3s:
    extraArgs:
      - arg: '--disable=traefik,servicelb'

1. Which is the difference between both config settings, options.k3d.disableLoadBalancer and options.k3s.extraArgs["--disable=servicelb"]? Why do I need to set both?

Also, I've not quite figure out how to configure k3d and metallb.

I mean, I've been able to configure metallb:

configInline:
  address-pools:
   - name: default
     protocol: layer2
     addresses:
     - "172.28.0.0/16"

pool ips are extracted from k3d docker network assigned to created cluster:

output of $ docker network inspect k3d-salut | jq 'map(.IPAM.Config | first) | first':

{
  "Subnet": "172.28.0.0/16",
  "Gateway": "172.28.0.1"
}

Nodes have assigned those ips:

Output of $ kubectl get -o=wide nodes

NAME                 STATUS   ROLES                  AGE   VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE   KERNEL-VERSION     CONTAINER-RUNTIME
k3d-salut-agent-0    Ready    <none>                 11h   v1.21.7+k3s1   172.28.0.2    <none>        Unknown    5.4.0-94-generic   containerd://1.4.12-k3s1
k3d-salut-agent-1    Ready    <none>                 11h   v1.21.7+k3s1   172.28.0.4    <none>        Unknown    5.4.0-94-generic   containerd://1.4.12-k3s1
k3d-salut-server-0   Ready    control-plane,master   11h   v1.21.7+k3s1   172.28.0.3    <none>        Unknown    5.4.0-94-generic   containerd://1.4.12-k3s1

Next question is:

2. Only one ip can be assigned as external ip by metallb since k3d-salut-server-0 have allocated a single ip? does it mean, I'm only be able to create on single LoadBalancer?

I mean, when I'm trying to get access from my host to my cluster, I'm only be able to reach it using k3d-salut-server-0 assigned ip, isn't it? why metallb CIDR has to be 172.28.0.0/16?

I mean, if I deploy four LoadBalancer, metallb will get me 172.28.0.0, 172.28.0.1, 172.28.0.2, 172.28.0.3, but only one of them can be reached from my host, 172.28.0.3. Is it right?

[iwilltry42]

Which is the difference between both config settings, options.k3d.disableLoadBalancer and options.k3s.extraArgs["--disable=servicelb"]? Why do I need to set both?

As you can see from the config, one is for/from k3d and the other's for/from K3s:

• options.k3d.disableLoadBalancer disables the k3d loadbalancer (k3d-<cluster>-serverlb docker container running next to your node containers), which proxies traffic from the outside into your cluster
• options.k3s.extraArgs["--disable=servicelb"] disables the servicelb (klipper) inside the cluster which is used to support LoadBalancer type services inside the cluster possible

Only one ip can be assigned as external ip by metallb since k3d-salut-server-0 have allocated a single ip? does it mean, I'm only be able to create on single LoadBalancer?
I mean, when I'm trying to get access from my host to my cluster, I'm only be able to reach it using k3d-salut-server-0 assigned ip, isn't it? why metallb CIDR has to be 172.28.0.0/16?

I cannot tell you much on MetalLB, as it's unrelated to k3d, but afaik MetalLB just assigns IPs from any pool that you give it.
I guess that giving it the IP Pool of the docker network could work, since your host should already know how to route traffic there via the docker bridge interface.. but how would you configure it to be routed to the node containers and then into the cluster?
What's reachable and what not depends on your setup, I cannot tell from the above information how you set everything up and how your environment and host system look like.
FWIW: reaching the IPs inside the docker network will only work on Linux systems, as on Docker for Mac and Docker for Windows, the containers are inside the Docker VM.

Here are some resources on MetalLB and k3d that google brought up:

• https://github.com/keunlee/k3d-metallb-starter-kit
• https://blog.kubernauts.io/k3s-with-k3d-and-metallb-on-mac-923a3255c36e
• https://dzone.com/articles/how-to-create-a-kubernetes-cluster-and-load-balanc

Are you really strongly bound to using LoadBalancer in your development setup with k3d?
The concept is mostly for production uses where you need externally accessible IPs 🤔