What happened?

We run periodic conformance tests in github.com/kubernetes-sigs/cluster-api and they started to fail https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api#capi-e2e-latestk8s-main

These currently

• use https://dl.k8s.io/ci/latest-1.33.txt to build kind images from source with that version
• create a Cluster with that version by using Cluster API and cluster-api-provider-docker (and in this case with dualstack ip enabled)
• They set --service-cluster-ip-range=10.128.0.0/12,fd00:100:64::/108 in kube-apiserver (and kube-controller-manager)
• And after finishing cluster creation run conformance tests, including the selector for the IPv6DualStack feature tests.

Several conformace tests for IPv6DualStack then failed with e.g. the following message:

I0124 07:48:52.822155 16 utils.go:776] Unexpected error: Failed to create node-port-service service: Internal error occurred: failed to allocate a serviceIP: range is full: 
    <*errors.StatusError | 0xc0031bb9a0>: 
    Internal error occurred: failed to allocate a serviceIP: range is full
    {
        ErrStatus: 
            code: 500
            details:
              causes:
              - message: 'failed to allocate a serviceIP: range is full'
            message: 'Internal error occurred: failed to allocate a serviceIP: range is full'
            metadata: {}
            reason: InternalError
            status: Failure,
    }
[FAILED] Failure recorded during attempt 1:

Note: The kube-apiserver is able to create the IPAddress object, but the service does not get created.

What did you expect to happen?

Tests to succeed.

How can we reproduce it (as minimally and precisely as possible)?

Build kindest/node image for k/k commit 0798325

# checkout k/k on 0798325ba13643
kind build node-image --image "kindest/node:v1.33.0-alpha.0.614_0798325ba13643" $(pwd)

Create a kind cluster using the following configuration:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  ipFamily: dual
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    apiServer:
        extraArgs:
          service-cluster-ip-range: "10.96.0.10/12,fd00:100:64::/108"

kind create cluster --name test --config kind-cluster.yaml --image kindest/node:v1.33.0-alpha.0.614_0798325ba13643

Try to create a service with .spec.ipFamilyPolicy set to RequireDualStack or .spec.ipFamilies set to ["IPv6"]:

$ kubectl create svc clusterip --tcp=80:80 foo -o json --dry-run=client | jq '.spec.ipFamilies = ["IPv6"]' | k apply -f -
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed to allocate a serviceIP: range is full

$ kubectl create svc clusterip --tcp=80:80 foo -o json --dry-run=client | jq '.spec.ipFamilies = ["IPv6"]' | k apply -f -
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed to allocate a serviceIP: range is full

Note: there's a chance that the ip allocation in kube-apiserver picks an ipv6 address which is then part the first /112 part of and then works.

Anything else we need to know?

Follow-up to #129753 (comment)

I also stepped into kube-apiserver:

• It fails to validate here that the ip is contained in the CIDR (example configuration: CIDR fd00:100:64::/108, IP fd00:100:64::f:321c

  • kubernetes/pkg/registry/core/service/ipallocator/bitmap.go

    Lines 173 to 179 in 0798325

    	ok, offset := r.contains(ip)
    	if !ok {
    	if !dryRun {
    	// update metrics
    	r.metrics.incrementAllocationErrors(label.String(), "static")
    	}
    	return &ErrNotInRange{ip, r.net.String()}

• The IP is part of that network, but the bitmap is capped to /112

Workarounds:

• Enable FeatureGate DisableAllocatorDualWrite in kube-apiserver
• Use a /112 cidr for services

cc @aojea

And Kudos for Antonio helping me to dig into it!

Kubernetes version

❯ kubectl version
Client Version: v1.29.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.33.0-alpha.0.614+0798325ba13643
WARNING: version difference between client (1.29) and server (1.33) exceeds the supported minor version skew of +/-1

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)