Fails to build 1.21.0 node image - `docker buildx` now required #2188

Arnavion · 2021-04-08T19:22:41Z

What happened:

rm -rf '/tmp/kubernetes-v1.21.0'
git clone --recurse-submodules --branch=v1.21 --depth=1 'https://github.com/kubernetes/kubernetes' '/tmp/kubernetes-v1.21.0'
kind build node-image --image 'kindest/node:v1.21.0' --kube-root '/tmp/kubernetes-v1.21.0'

The kind command fails with:

Click here

Starting to build Kubernetes
+++ [0408 12:06:17] Verifying Prerequisites....
+++ [0408 12:06:17] Building Docker image kube-build:build-8406c34423-5-v1.16.1-1
+++ [0408 12:11:29] Creating data container kube-build-data-8406c34423-5-v1.16.1-1
+++ [0408 12:11:30] Syncing sources to container
+++ [0408 12:11:34] Output from this container will be rsynced out upon completion. Set KUBE_RUN_COPY_OUTPUT=n to disable.
+++ [0408 12:11:34] Running build command...
+++ [0408 12:11:42] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/prerelease-lifecycle-gen
Generating prerelease lifecycle code for 27 targets
+++ [0408 12:11:45] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/deepcopy-gen
Generating deepcopy code for 227 targets
+++ [0408 12:11:52] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/defaulter-gen
Generating defaulter code for 89 targets
+++ [0408 12:12:00] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/conversion-gen
Generating conversion code for 124 targets
+++ [0408 12:12:18] Building go targets for linux/amd64:
    ./vendor/k8s.io/kube-openapi/cmd/openapi-gen
Generating openapi code for KUBE
Generating openapi code for AGGREGATOR
Generating openapi code for APIEXTENSIONS
Generating openapi code for CODEGEN
Generating openapi code for SAMPLEAPISERVER
+++ [0408 12:12:29] Building go targets for linux/amd64:
    ./vendor/github.com/go-bindata/go-bindata/go-bindata
+++ [0408 12:12:30] Building go targets for linux/amd64:
    cmd/kubeadm
    cmd/kubectl
    cmd/kubelet
+++ [0408 12:13:06] Syncing out of container
+++ [0408 12:13:08] Verifying Prerequisites....
+++ [0408 12:13:09] Building Docker image kube-build:build-8406c34423-5-v1.16.1-1
+++ [0408 12:13:18] Syncing sources to container
+++ [0408 12:13:21] Running build command...
Generating prerelease lifecycle code for 27 targets
Generating deepcopy code for 227 targets
Generating defaulter code for 89 targets
Generating conversion code for 124 targets
Generating openapi code for KUBE
Generating openapi code for AGGREGATOR
Generating openapi code for APIEXTENSIONS
Generating openapi code for CODEGEN
Generating openapi code for SAMPLEAPISERVER
+++ [0408 12:14:06] Building go targets for linux/amd64:
    cmd/kube-apiserver
    cmd/kube-controller-manager
    cmd/kube-scheduler
    cmd/kube-proxy
+++ [0408 12:14:24] Syncing out of container
+++ [0408 12:14:27] Building images: linux-amd64
+++ [0408 12:14:27] Starting docker build for image: kube-apiserver-amd64
+++ [0408 12:14:27] Starting docker build for image: kube-controller-manager-amd64
+++ [0408 12:14:27] Starting docker build for image: kube-scheduler-amd64
+++ [0408 12:14:27] Starting docker build for image: kube-proxy-amd64
unknown shorthand flag: 'f' in -f
See 'docker --help'.

Usage:  docker [OPTIONS] COMMAND

A self-sufficient runtime for containers

Options:
      --config string      Location of client config files (default
                           "/home/arnavion/.docker")
  -c, --context string     Name of the context to use to connect to the
                           daemon (overrides DOCKER_HOST env var and
                           default context set with "docker context use")
  -D, --debug              Enable debug mode
  -H, --host list          Daemon socket(s) to connect to
  -l, --log-level string   Set the logging level
                           ("debug"|"info"|"warn"|"error"|"fatal")
                           (default "info")
      --tls                Use TLS; implied by --tlsverify
      --tlscacert string   Trust certs signed only by this CA (default
                           "/home/arnavion/.docker/ca.pem")
      --tlscert string     Path to TLS certificate file (default
                           "/home/arnavion/.docker/cert.pem")
      --tlskey string      Path to TLS key file (default
                           "/home/arnavion/.docker/key.pem")
      --tlsverify          Use TLS and verify the remote
  -v, --version            Print version information and quit

[...]

Something is calling docker -f ... , preumably it meant to call docker <subcommand> -f

What you expected to happen:

Expected it to succeed.

How to reproduce it (as minimally and precisely as possible):

As above.

Anything else we need to know?:

This works with other Kubernetes versions from 1.14 through 1.20, so this is something 1.21-specific.

Environment:

kind version: (use kind version): kind v0.10.0 go1.15.7 linux/amd64
Kubernetes version: (use kubectl version): 1.21.0
Docker version: (use docker info): Server Version: 20.10.5-ce
OS (e.g. from /etc/os-release): openSUSE Tumbleweed 20210330

Edit: strace indicates the problematic command is:

[pid 83376] execve("/usr/bin/docker", ["docker", "buildx", "build", "-f", "/tmp/kubernetes-v1.21.0/build/server-image/kube-apiserver/Dockerfile", "--platform", "linux/amd64", "--load", "--pull", "-t", "k8s.gcr.io/kube-apiserver-amd64:v1.21.0", "--build-arg", "BASEIMAGE=k8s.gcr.io/build-image/go-runner:v2.3.1-go1.16.1-buster.0", "--build-arg", "SETCAP_IMAGE=k8s.gcr.io/build-image/setcap:buster-v1.4.0", "--build-arg", "BINARY=kube-apiserver", "/tmp/kubernetes-v1.21.0/_output/release-stage/server/linux-amd64/kubernetes/server/bin/kube-apiserver.dockerbuild"], 0x556108f33680 /* 125 vars */ <unfinished ...>

ie

docker buildx build \
    -f '/tmp/kubernetes-v1.21.0/build/server-image/kube-apiserver/Dockerfile' \
    --platform linux/amd64 \
    --load \
    --pull \
    -t 'k8s.gcr.io/kube-apiserver-amd64:v1.21.0' \
    --build-arg 'BASEIMAGE=k8s.gcr.io/build-image/go-runner:v2.3.1-go1.16.1-buster.0' \
    --build-arg 'SETCAP_IMAGE=k8s.gcr.io/build-image/setcap:buster-v1.4.0' \
    --build-arg 'BINARY=kube-apiserver' \
    '/tmp/kubernetes-v1.21.0/_output/release-stage/server/linux-amd64/kubernetes/server/bin/kube-apiserver.dockerbuild'

Running that command by itself also has the same problem.

Edit 2: Okay, the problem is my distro-provided docker CLI does not have the buildx plugin. docker foo build -f produces this confusing error when foo can't be found as a plugin.

Is this something that can be fixed by kind (given it's new with 1.21 and was not needed for 1.20 and earlier) ? If not, it needs to be documented somewhere that this docker CLI plugin is required to build node images.

The text was updated successfully, but these errors were encountered:

neolit123 · 2021-04-08T21:39:29Z

Edit 2: Okay, the problem is my distro-provided docker CLI does not have the buildx plugin. docker foo build -f produces this confusing error when foo can't be found as a plugin.

yes, the plugin is required.
kind executes the kubernetes/kuberentes scripts for building the images and they use buildx.
https://github.com/kubernetes/kubernetes/search?q=buildx

Is this something that can be fixed by kind (given it's new with 1.21 and was not needed for 1.20 and earlier) ? If not, it needs to be documented somewhere that this docker CLI plugin is required to build node images.

i think for the time being the requirement should be documented in the kind node image docs.

neolit123 · 2021-04-08T21:39:47Z

/remove-kind bug
/kind documentation

Arnavion · 2021-04-08T21:40:27Z

Okay, so once I installed the buildx plugin I was able to build the node image, however it fails to start:

Click here

$ kind create cluster --name v1.21 --image 'kindest/node:v1.21'

Loaded image: kindest/node:v1.21.0
Creating cluster "v1.21" ...
 ✓ Ensuring node image (kindest/node:v1.21.0) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✗ Starting control-plane 🕹️
ERROR: failed to create cluster: failed to init node with kubeadm: command "docker exec --privileged v1.21-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1
Command Output: I0408 21:33:41.777965     248 initconfiguration.go:246] loading configuration from "/kind/kubeadm.conf"
[config] WARNING: Ignored YAML document with GroupVersionKind kubeadm.k8s.io/v1beta2, Kind=JoinConfiguration
I0408 21:33:41.789251     248 kubelet.go:259] setting the KubeletConfiguration cgroupDriver to "systemd"
[init] Using Kubernetes version: v1.21.0
[certs] Using certificateDir folder "/etc/kubernetes/pki"
I0408 21:33:41.789357     248 certs.go:110] creating a new certificate authority for ca
[certs] Generating "ca" certificate and key
I0408 21:33:41.957924     248 certs.go:487] validating certificate period for ca certificate
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost v1.21-control-plane] and IPs [10.96.0.1 172.18.0.2 127.0.0.1]
I0408 21:33:42.469189     248 certs.go:110] creating a new certificate authority for front-proxy-ca
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
I0408 21:33:42.602751     248 certs.go:487] validating certificate period for front-proxy-ca certificate
[certs] Generating "front-proxy-client" certificate and key
I0408 21:33:42.757067     248 certs.go:110] creating a new certificate authority for etcd-ca
[certs] Generating "etcd/ca" certificate and key
I0408 21:33:42.908739     248 certs.go:487] validating certificate period for etcd/ca certificate
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost v1.21-control-plane] and IPs [172.18.0.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost v1.21-control-plane] and IPs [172.18.0.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
I0408 21:33:43.599110     248 certs.go:76] creating new public/private key files for signing service account users
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0408 21:33:43.927559     248 kubeconfig.go:101] creating kubeconfig file for admin.conf
[kubeconfig] Writing "admin.conf" kubeconfig file
I0408 21:33:44.299408     248 kubeconfig.go:101] creating kubeconfig file for kubelet.conf
[kubeconfig] Writing "kubelet.conf" kubeconfig file
I0408 21:33:44.716312     248 kubeconfig.go:101] creating kubeconfig file for controller-manager.conf
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0408 21:33:44.875861     248 kubeconfig.go:101] creating kubeconfig file for scheduler.conf
[kubeconfig] Writing "scheduler.conf" kubeconfig file
I0408 21:33:45.208758     248 kubelet.go:63] Stopping the kubelet
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
I0408 21:33:45.291372     248 manifests.go:96] [control-plane] getting StaticPodSpecs
I0408 21:33:45.291840     248 certs.go:487] validating certificate period for CA certificate
I0408 21:33:45.291943     248 manifests.go:109] [control-plane] adding volume "ca-certs" for component "kube-apiserver"
I0408 21:33:45.291954     248 manifests.go:109] [control-plane] adding volume "etc-ca-certificates" for component "kube-apiserver"
I0408 21:33:45.291960     248 manifests.go:109] [control-plane] adding volume "k8s-certs" for component "kube-apiserver"
I0408 21:33:45.291966     248 manifests.go:109] [control-plane] adding volume "usr-local-share-ca-certificates" for component "kube-apiserver"
I0408 21:33:45.291974     248 manifests.go:109] [control-plane] adding volume "usr-share-ca-certificates" for component "kube-apiserver"
I0408 21:33:45.302742     248 manifests.go:126] [control-plane] wrote static Pod manifest for component "kube-apiserver" to "/etc/kubernetes/manifests/kube-apiserver.yaml"
I0408 21:33:45.302766     248 manifests.go:96] [control-plane] getting StaticPodSpecs
[control-plane] Creating static Pod manifest for "kube-controller-manager"
I0408 21:33:45.303107     248 manifests.go:109] [control-plane] adding volume "ca-certs" for component "kube-controller-manager"
I0408 21:33:45.303119     248 manifests.go:109] [control-plane] adding volume "etc-ca-certificates" for component "kube-controller-manager"
I0408 21:33:45.303125     248 manifests.go:109] [control-plane] adding volume "flexvolume-dir" for component "kube-controller-manager"
I0408 21:33:45.303132     248 manifests.go:109] [control-plane] adding volume "k8s-certs" for component "kube-controller-manager"
I0408 21:33:45.303138     248 manifests.go:109] [control-plane] adding volume "kubeconfig" for component "kube-controller-manager"
I0408 21:33:45.303144     248 manifests.go:109] [control-plane] adding volume "usr-local-share-ca-certificates" for component "kube-controller-manager"
I0408 21:33:45.303151     248 manifests.go:109] [control-plane] adding volume "usr-share-ca-certificates" for component "kube-controller-manager"
I0408 21:33:45.304293     248 manifests.go:126] [control-plane] wrote static Pod manifest for component "kube-controller-manager" to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
I0408 21:33:45.304310     248 manifests.go:96] [control-plane] getting StaticPodSpecs
[control-plane] Creating static Pod manifest for "kube-scheduler"
I0408 21:33:45.304601     248 manifests.go:109] [control-plane] adding volume "kubeconfig" for component "kube-scheduler"
I0408 21:33:45.305220     248 manifests.go:126] [control-plane] wrote static Pod manifest for component "kube-scheduler" to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0408 21:33:45.306161     248 local.go:74] [etcd] wrote Static Pod manifest for a local etcd member to "/etc/kubernetes/manifests/etcd.yaml"
I0408 21:33:45.306176     248 waitcontrolplane.go:87] [wait-control-plane] Waiting for the API server to be healthy
I0408 21:33:45.307220     248 loader.go:372] Config loaded from file:  /etc/kubernetes/admin.conf
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0408 21:33:45.308956     248 round_trippers.go:454] GET https://v1.21-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I0408 21:33:45.810371     248 round_trippers.go:454] GET https://v1.21-control-plane:6443/healthz?timeout=10s  in 0 milliseconds

[...]

I0408 21:34:24.809786     248 round_trippers.go:454] GET https://v1.21-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
I0408 21:34:25.310322     248 round_trippers.go:454] GET https://v1.21-control-plane:6443/healthz?timeout=10s  in 0 milliseconds

[...]

I0408 21:35:40.310530     248 round_trippers.go:454] GET https://v1.21-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.

        Unfortunately, an error has occurred:
                timed out waiting for the condition

        This error is likely caused by:
                - The kubelet is not running
                - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

        If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
                - 'systemctl status kubelet'
                - 'journalctl -xeu kubelet'

        Additionally, a control plane component may have crashed or exited when started by the container runtime.
        To troubleshoot, list all containers using your preferred container runtimes CLI.

        Here is one example how you may list all Kubernetes containers running in cri-o/containerd using crictl:
                - 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
                Once you have found the failing container, you can inspect its logs with:
                - 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock logs CONTAINERID'

couldn't initialize a Kubernetes cluster
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init.runWaitControlPlanePhase
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init/waitcontrolplane.go:114
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:234
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:421
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/init.go:152
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:850
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:958
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:895
k8s.io/kubernetes/cmd/kubeadm/app.Run
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50
main.main
        _output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
runtime.main
        /usr/local/go/src/runtime/proc.go:225
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1371
error execution phase wait-control-plane
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:235
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:421
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/init.go:152
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:850
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:958
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:895
k8s.io/kubernetes/cmd/kubeadm/app.Run
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50
main.main
        _output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
runtime.main
        /usr/local/go/src/runtime/proc.go:225
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1371

journalctl inside the container reveals:

Apr 08 21:27:42 v1.21-control-plane systemd[1]: Starting kubelet: The Kubernetes Node Agent...
Apr 08 21:27:42 v1.21-control-plane systemd[1]: Started kubelet: The Kubernetes Node Agent.
Apr 08 21:27:42 v1.21-control-plane kubelet[1463]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 08 21:27:42 v1.21-control-plane kubelet[1463]: Flag --provider-id has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 08 21:27:42 v1.21-control-plane kubelet[1463]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 08 21:27:42 v1.21-control-plane kubelet[1463]: Flag --cgroup-root has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 08 21:27:42 v1.21-control-plane kubelet[1463]: I0408 21:27:42.485221    1463 server.go:197] "Warning: For remote container runtime, --pod-infra-container-image is ignored in kubelet, which should be set in that remote runtime instead"
Apr 08 21:27:42 v1.21-control-plane kubelet[1463]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 08 21:27:42 v1.21-control-plane kubelet[1463]: Flag --provider-id has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 08 21:27:42 v1.21-control-plane kubelet[1463]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 08 21:27:42 v1.21-control-plane kubelet[1463]: Flag --cgroup-root has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 08 21:27:42 v1.21-control-plane systemd[1]: Started Kubernetes systemd probe.
Apr 08 21:27:42 v1.21-control-plane kubelet[1463]: I0408 21:27:42.499384    1463 server.go:440] "Kubelet version" kubeletVersion="v1.21.0"
Apr 08 21:27:42 v1.21-control-plane kubelet[1463]: I0408 21:27:42.499704    1463 server.go:851] "Client rotation is on, will bootstrap in background"
Apr 08 21:27:42 v1.21-control-plane kubelet[1463]: I0408 21:27:42.502285    1463 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Apr 08 21:27:42 v1.21-control-plane kubelet[1463]: I0408 21:27:42.503416    1463 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/pki/ca.crt
Apr 08 21:27:42 v1.21-control-plane systemd[1]: run-r2444a3a0313e45eda80ef28542432b06.scope: Succeeded.
Apr 08 21:27:42 v1.21-control-plane kubelet[1463]: E0408 21:27:42.537150    1463 server.go:292] "Failed to run kubelet" err="failed to run Kubelet: invalid configuration: cgroup-root [\"kubelet\"] doesn't exist"
Apr 08 21:27:42 v1.21-control-plane systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Apr 08 21:27:42 v1.21-control-plane systemd[1]: kubelet.service: Failed with result 'exit-code'.
Apr 08 21:27:43 v1.21-control-plane systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 41.
Apr 08 21:27:43 v1.21-control-plane systemd[1]: Stopped kubelet: The Kubernetes Node Agent.

neolit123 · 2021-04-08T21:47:57Z

related to #1969
see some of the linked kubernetes/kubernetes issues in there.

one points a problem with the host OS.
for another user setting an explicit cgroupRoot worked.

Arnavion · 2021-04-08T21:56:10Z

I don't have this issue deploying a v1.20 cluster with kind 0.10.0, for which the --cgroup-root=/kubelet parameter is also given to kubelet.

Arnavion · 2021-04-08T22:14:59Z

I guess v1.21 changed to require cgroups v2? That's the only thing that makes sense.

If that's the case, then yes my distro doesn't have the file it's looking for. The kubelet service unit has an ExecStartPre (injected via /etc/systemd/system/kubelet.service.d/10-kubeadm.conf ):

ExecStartPre=/bin/sh -euc "if [ -f /sys/fs/cgroup/cgroup.controllers ]; then create-kubelet-cgroup-v2; fi"

which means the script is entirely skipped on cgroup v1 hosts. If the script was run, it would've produced a more descriptive error:

if [[ ! -f "/sys/fs/cgroup/cgroup.controllers" ]]; then
        echo 'ERROR: this script should not be called on cgroup v1 hosts' >&2
        exit 1
fi

If v1.21 requires cgroups v2, it would be useful to run that script unguarded so that the error can be seen in the journal.

neolit123 · 2021-04-08T22:19:49Z

could you please log a separate issue and provide the details there?
we can keep this one for the node image documentation.

Arnavion · 2021-04-08T22:34:36Z

Done. #2189

BenTheElder · 2021-04-09T00:06:26Z

1.21 requires KIND @ HEAD due to the breaking change in upstream Kubernetes.

Similarly the build requirement for buildx comes entirely from Kubernetes upstream.

Cgroups v2 is not required, and that script is only for v2.

BenTheElder · 2021-04-09T00:07:40Z

The breaking cgroups change is kubeadm defaulting kubelet to systemd cgroup driver instead of cgroupsfs in 1.21 which is an action required change for cluster administrators.

Kind handles this, but not in v0.10 as when v0.10 was released this was not necessary / known.

BenTheElder · 2021-04-09T00:08:30Z

In general if you wish to run Kubernetes newer than the kind binary you are using it may not work due to changes like this. We will be releasing in the next week or so after #2176

jason-kane · 2021-04-26T04:33:31Z

Was there an unexpected issue with 2176? If a release with 1.21 support going to be a while longer, is HEAD likely to work w/1.21 on non-ARM?

BenTheElder · 2021-04-27T18:03:53Z

I've had some personal / work reasons that I've sort of just been responding to issues / discussions and keeping after reviews, not working on bug fixes / features for a little bit. I'll be back to it.
HEAD does work well, hopefully won't be much longer.

Also the next round of upstream Kubernetes patch releases will fix a major regression in 1.18.x ... 1.21.x where the current patch releases startup significantly slower than before, so we'd kinda like to release alongside those.

markusthoemmes · 2021-05-04T15:29:54Z

In case anybody reaches here trying to build the image with podman, here's the necessary diff for K8s

diff --git a/build/lib/release.sh b/build/lib/release.sh
index d8fb1f3df2f..8226d826e44 100644
--- a/build/lib/release.sh
+++ b/build/lib/release.sh
@@ -379,10 +379,10 @@ function kube::release::create_docker_images_for_server() {
         ln "${binary_file_path}" "${docker_build_path}/${binary_name}"
 
         local build_log="${docker_build_path}/build.log"
-        if ! DOCKER_CLI_EXPERIMENTAL=enabled "${DOCKER[@]}" buildx build \
+        if ! DOCKER_CLI_EXPERIMENTAL=enabled "${DOCKER[@]}" build \
           -f "${docker_file_path}" \
           --platform linux/"${arch}" \
-          --load ${docker_build_opts:+"${docker_build_opts}"} \
+          ${docker_build_opts:+"${docker_build_opts}"} \
           -t "${docker_image_tag}" \
           --build-arg BASEIMAGE="${base_image}" \
           --build-arg SETCAP_IMAGE="${KUBE_BUILD_SETCAP_IMAGE}" \

Gonna investigate if that upstream code can be made somewhat more flexible to support podman (and maybe even non buildx docker, if builds of only the "local" arch are necessary)

BenTheElder · 2021-05-04T16:49:15Z

Buildx is used to ensure extended permissions are persisted and is necessary for the migration towards a rootless control plane. It is also helpful for ensuring cross complication produces correct image metadata.

The dockerfiles also leverage env it populates to streamline them.

I am a maintainer of the upstream build and can confidently say we're not interested in losing that, the Kubernetes build process requires a recent install of docker. It also requires a recent bash, and gnu coreutils instead of bsd / Darwin. It doesn't work on windows either. It's just not practical to do what we need otherwise.

BenTheElder · 2021-05-04T17:51:17Z

See for example past discussion here including another maintainer at RedHat: kubernetes/kubernetes#79048 (comment)

This is also a little off-topic for this repo though and discussing upstream build changes upstream will bring visibility to more of the maintainers.

This issue here is only open until we make a clear note in our own docs that Kubernetes's source build dependencies are defined by upstream and pointing to the upstream docs for this.

jason-kane · 2021-05-04T18:23:04Z

I can confirm that HEAD kind appears to work flawlessly with k8s 1.21

markusthoemmes · 2021-05-05T06:34:48Z

@BenTheElder understood, thanks for the response! I just left this here in case anybody else was stumbling and needed a potential workaround.

Arnavion added the kind/bug Categorizes issue or PR as related to a bug. label Apr 8, 2021

Arnavion mentioned this issue Apr 8, 2021

1.21.0 Arnavion/k8s-openapi#89

Closed

k8s-ci-robot added kind/documentation Categorizes issue or PR as related to documentation. and removed kind/bug Categorizes issue or PR as related to a bug. labels Apr 8, 2021

Arnavion changed the title ~~Fails to build 1.21.0 node image~~ Fails to build 1.21.0 node image - docker buildx now required Apr 8, 2021

BenTheElder mentioned this issue Apr 9, 2021

Fails to start 1.21.0 cluster - kubelet complains about missing cgroup #2189

Closed

ckotzbauer mentioned this issue Apr 12, 2021

Update test matrix to latest 3 sets of k8s releases kubereboot/kured#349

Merged

sbueringer mentioned this issue Apr 12, 2021

Periodic e2e test for workload cluster upgrade from v1.21 to latest are failing kubernetes-sigs/cluster-api#4462

Closed

BenTheElder mentioned this issue Apr 13, 2021

docs: drop bazel #2192

Merged

evrardjp mentioned this issue Apr 28, 2021

Update k8s test matrix to include 1.{19,20,21} kubereboot/kured#347

Closed

BenTheElder mentioned this issue May 18, 2021

cover upstream build requirements #2253

Merged

BenTheElder self-assigned this May 18, 2021

k8s-ci-robot closed this as completed in #2253 May 18, 2021

kashifest mentioned this issue May 18, 2021

Uplift kind node image and kind binary metal3-io/metal3-dev-env#658

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fails to build 1.21.0 node image - `docker buildx` now required #2188

Fails to build 1.21.0 node image - `docker buildx` now required #2188

Arnavion commented Apr 8, 2021 •

edited

Loading

neolit123 commented Apr 8, 2021

neolit123 commented Apr 8, 2021

Arnavion commented Apr 8, 2021

neolit123 commented Apr 8, 2021

Arnavion commented Apr 8, 2021

Arnavion commented Apr 8, 2021

neolit123 commented Apr 8, 2021 •

edited

Loading

Arnavion commented Apr 8, 2021

BenTheElder commented Apr 9, 2021

BenTheElder commented Apr 9, 2021 •

edited

Loading

BenTheElder commented Apr 9, 2021

jason-kane commented Apr 26, 2021

BenTheElder commented Apr 27, 2021

markusthoemmes commented May 4, 2021

BenTheElder commented May 4, 2021

BenTheElder commented May 4, 2021

jason-kane commented May 4, 2021

markusthoemmes commented May 5, 2021

Fails to build 1.21.0 node image - docker buildx now required #2188

Fails to build 1.21.0 node image - docker buildx now required #2188

Comments

Arnavion commented Apr 8, 2021 • edited Loading

neolit123 commented Apr 8, 2021

neolit123 commented Apr 8, 2021

Arnavion commented Apr 8, 2021

neolit123 commented Apr 8, 2021

Arnavion commented Apr 8, 2021

Arnavion commented Apr 8, 2021

neolit123 commented Apr 8, 2021 • edited Loading

Arnavion commented Apr 8, 2021

BenTheElder commented Apr 9, 2021

BenTheElder commented Apr 9, 2021 • edited Loading

BenTheElder commented Apr 9, 2021

jason-kane commented Apr 26, 2021

BenTheElder commented Apr 27, 2021

markusthoemmes commented May 4, 2021

BenTheElder commented May 4, 2021

BenTheElder commented May 4, 2021

jason-kane commented May 4, 2021

markusthoemmes commented May 5, 2021

Fails to build 1.21.0 node image - `docker buildx` now required #2188

Fails to build 1.21.0 node image - `docker buildx` now required #2188

Arnavion commented Apr 8, 2021 •

edited

Loading

neolit123 commented Apr 8, 2021 •

edited

Loading

BenTheElder commented Apr 9, 2021 •

edited

Loading