Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Trying to use local registry need to ignore TLS or update containerd local registry mirror type. #3087

Closed
kcirtapfromspace opened this issue Feb 3, 2023 · 1 comment
Closed

Trying to use local registry need to ignore TLS or update containerd local registry mirror type. #3087

kcirtapfromspace opened this issue Feb 3, 2023 · 1 comment
Assignees
@BenTheElder
Labels
kind/support Categorizes issue or PR as a support question.

Comments

@kcirtapfromspace
Copy link

What happened:
Trying to have argo workflows to pull from local registry instead of reaching out to remote registry. I can see the tagged image is available. Most pods can be created; when argo create a pod from a workflow it seems to try to pull the image with https. Trying to configure the mirror to ignore tls.
pod:error.

Failed to pull image "kind-registry:5000/py_app": rpc error: code = Unknown desc = failed to pull and unpack image "kind-registry:5000/py_app:latest": failed to resolve reference "kind-registry:5000/py_app:latest": failed to do request: Head "https://kind-registry:5000/v2/py_app/manifests/latest": http: server gave HTTP response to HTTPS client

local registry:

kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}" |\
tr -s '[[:space:]]' '\n' |\
sort |\
uniq -c
   1 docker.io/kindest/kindnetd:v20221004-44d545d1
   1 docker.io/kindest/local-path-provisioner:v0.0.22-kind.0
   1 kind-registry:5000/py_app
   1 localhost:5001/py_app:tilt-187a11815517159f
   1 postgres:12-alpine
   1 quay.io/argoproj/argocli:v3.4.4
   1 quay.io/argoproj/argoexec:v3.4.4
   1 quay.io/argoproj/workflow-controller:v3.4.4
   2 registry.k8s.io/coredns/coredns:v1.9.3
   1 registry.k8s.io/etcd:3.5.4-0
   1 registry.k8s.io/kube-apiserver:v1.25.3
   1 registry.k8s.io/kube-controller-manager:v1.25.3
   1 registry.k8s.io/kube-proxy:v1.25.3
   1 registry.k8s.io/kube-scheduler:v1.25.3

What should be cleaned up or changed:
https://kind.sigs.k8s.io/docs/user/local-registry/
the script generally works, but no longer reflects guidance for using containerd.
specifically this section is out of date.

# create a cluster with the local registry enabled in containerd
cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:${reg_port}"]
    endpoint = ["http://${reg_name}:5000"]
EOF

https://github.com/containerd/containerd/blob/release/1.6/docs/hosts.md
Make it easier for developers to get started without needing to chase down issue.

https://github.com/containerd/containerd/blob/release/1.6/docs/hosts.md
Why is this needed:

my first attempt did not go so well:

#!/bin/sh
set -o errexit

# create registry container unless it already exists
reg_name='kind-registry'
reg_port='5001'
if [ "$(docker inspect -f '{{.State.Running}}' "${reg_name}" 2>/dev/null || true)" != 'true' ]; then
    docker run \
        -d --restart=always -p "127.0.0.1:${reg_port}:5000" --name "${reg_name}"\
    registry:2
fi

# create a cluster with the local registry enabled in containerd
cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane

  extraMounts:
  - hostPath: /Users/thinkmac/kind-k8s-storage
    containerPath: /data
containerdConfigPatches:
- |-
    [plugins."io.containerd.grpc.v1.cri".registry]
        config_path = "/etc/containerd/certs.d"
EOF

mkdir -p /etc/containerd/certs.d/docker.io
cat <<EOF > /etc/containerd/certs.d/docker.io/hosts.toml
server = "https://registry-1.docker.io"

[host."http://${reg_name}:${reg_port}"]
    capabilities = ["pull", "resolve"]
    skip_verify = true
EOF

# connect the registry to the cluster network if not already connected
if [ "$(docker inspect -f='{{json .NetworkSettings.Networks.kind}}' "${reg_name}")" = 'null' ]; then
    docker network connect "kind" "${reg_name}"
fi

# Document the local registry
# https://github.com/kubernetes/enhancements/tree/master/keps/sig-cluster-lifecycle/generic/1755-communicating-a-local-registry
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
    name: local-registry-hosting
    namespace: kube-public
data:
    localRegistryHosting.v1: |
        host: "localhost:${reg_port}"
        help: "https://kind.sigs.k8s.io/docs/user/local-registry/"
EOF

Error Log

❯ ~/kind-with-registry2.sh
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.25.3) 🖼
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✗ Starting control-plane 🕹️ 
ERROR: failed to create cluster: failed to init node with kubeadm: command "docker exec --privileged kind-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1
Command Output: I0203 22:47:47.677841     170 initconfiguration.go:254] loading configuration from "/kind/kubeadm.conf"
W0203 22:47:47.679000     170 initconfiguration.go:331] [config] WARNING: Ignored YAML document with GroupVersionKind kubeadm.k8s.io/v1beta3, Kind=JoinConfiguration
[init] Using Kubernetes version: v1.25.3
[certs] Using certificateDir folder "/etc/kubernetes/pki"
I0203 22:47:47.683072     170 certs.go:112] creating a new certificate authority for ca
[certs] Generating "ca" certificate and key
I0203 22:47:47.824172     170 certs.go:522] validating certificate period for ca certificate
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kind-control-plane kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost] and IPs [10.96.0.1 172.18.0.2 127.0.0.1]
[certs] Generating "apiserver-kubelet-client" certificate and key
I0203 22:47:48.139021     170 certs.go:112] creating a new certificate authority for front-proxy-ca
[certs] Generating "front-proxy-ca" certificate and key
I0203 22:47:48.209122     170 certs.go:522] validating certificate period for front-proxy-ca certificate
[certs] Generating "front-proxy-client" certificate and key
I0203 22:47:48.424819     170 certs.go:112] creating a new certificate authority for etcd-ca
[certs] Generating "etcd/ca" certificate and key
I0203 22:47:48.468848     170 certs.go:522] validating certificate period for etcd/ca certificate
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [kind-control-plane localhost] and IPs [172.18.0.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [kind-control-plane localhost] and IPs [172.18.0.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
I0203 22:47:48.847011     170 certs.go:78] creating new public/private key files for signing service account users
[certs] Generating "sa" key and public key
I0203 22:47:49.040295     170 kubeconfig.go:103] creating kubeconfig file for admin.conf
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
I0203 22:47:49.125516     170 kubeconfig.go:103] creating kubeconfig file for kubelet.conf
[kubeconfig] Writing "kubelet.conf" kubeconfig file
I0203 22:47:49.361677     170 kubeconfig.go:103] creating kubeconfig file for controller-manager.conf
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0203 22:47:49.471895     170 kubeconfig.go:103] creating kubeconfig file for scheduler.conf
[kubeconfig] Writing "scheduler.conf" kubeconfig file
I0203 22:47:49.507811     170 kubelet.go:66] Stopping the kubelet
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
I0203 22:47:49.581841     170 manifests.go:99] [control-plane] getting StaticPodSpecs
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
I0203 22:47:49.582097     170 certs.go:522] validating certificate period for CA certificate
I0203 22:47:49.582156     170 manifests.go:125] [control-plane] adding volume "ca-certs" for component "kube-apiserver"
I0203 22:47:49.582162     170 manifests.go:125] [control-plane] adding volume "etc-ca-certificates" for component "kube-apiserver"
I0203 22:47:49.582165     170 manifests.go:125] [control-plane] adding volume "k8s-certs" for component "kube-apiserver"
I0203 22:47:49.582168     170 manifests.go:125] [control-plane] adding volume "usr-local-share-ca-certificates" for component "kube-apiserver"
I0203 22:47:49.582174     170 manifests.go:125] [control-plane] adding volume "usr-share-ca-certificates" for component "kube-apiserver"
I0203 22:47:49.584442     170 manifests.go:154] [control-plane] wrote static Pod manifest for component "kube-apiserver" to "/etc/kubernetes/manifests/kube-apiserver.yaml"
I0203 22:47:49.584463     170 manifests.go:99] [control-plane] getting StaticPodSpecs
[control-plane] Creating static Pod manifest for "kube-controller-manager"
I0203 22:47:49.584596     170 manifests.go:125] [control-plane] adding volume "ca-certs" for component "kube-controller-manager"
I0203 22:47:49.584606     170 manifests.go:125] [control-plane] adding volume "etc-ca-certificates" for component "kube-controller-manager"
I0203 22:47:49.584609     170 manifests.go:125] [control-plane] adding volume "flexvolume-dir" for component "kube-controller-manager"
I0203 22:47:49.584611     170 manifests.go:125] [control-plane] adding volume "k8s-certs" for component "kube-controller-manager"
I0203 22:47:49.584614     170 manifests.go:125] [control-plane] adding volume "kubeconfig" for component "kube-controller-manager"
I0203 22:47:49.584616     170 manifests.go:125] [control-plane] adding volume "usr-local-share-ca-certificates" for component "kube-controller-manager"
I0203 22:47:49.584619     170 manifests.go:125] [control-plane] adding volume "usr-share-ca-certificates" for component "kube-controller-manager"
I0203 22:47:49.584959     170 manifests.go:154] [control-plane] wrote static Pod manifest for component "kube-controller-manager" to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
I0203 22:47:49.584971     170 manifests.go:99] [control-plane] getting StaticPodSpecs
[control-plane] Creating static Pod manifest for "kube-scheduler"
I0203 22:47:49.585084     170 manifests.go:125] [control-plane] adding volume "kubeconfig" for component "kube-scheduler"
I0203 22:47:49.585387     170 manifests.go:154] [control-plane] wrote static Pod manifest for component "kube-scheduler" to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0203 22:47:49.585789     170 local.go:65] [etcd] wrote Static Pod manifest for a local etcd member to "/etc/kubernetes/manifests/etcd.yaml"
I0203 22:47:49.585810     170 waitcontrolplane.go:83] [wait-control-plane] Waiting for the API server to be healthy
I0203 22:47:49.586127     170 loader.go:374] Config loaded from file:  /etc/kubernetes/admin.conf
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0203 22:47:49.587910     170 round_trippers.go:553] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
...
I0203 22:48:29.091623     170 round_trippers.go:553] GET https://kind-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
[kubelet-check] Initial timeout of 40s passed.
I0203 22:48:29.589425     170 round_trippers.go:553] GET https://kind-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
I0203 22:48:30.087884     170 round_trippers.go:553] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
...
I0203 22:48:34.590297     170 round_trippers.go:553] GET https://kind-control-plane:6443/healthz?timeout=10s  in 2 milliseconds
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
I0203 22:48:35.088056     170 round_trippers.go:553] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
...
I0203 22:48:44.590248     170 round_trippers.go:553] GET https://kind-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
I0203 22:48:45.088140     170 round_trippers.go:553] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
...
I0203 22:49:44.587926     170 round_trippers.go:553] GET https://kind-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.

Unfortunately, an error has occurred:
        timed out waiting for the condition

This error is likely caused by:
        - The kubelet is not running
        - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
        - 'systemctl status kubelet'
        - 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all running Kubernetes containers by using crictl:
        - 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
        Once you have found the failing container, you can inspect its logs with:
        - 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock logs CONTAINERID'
couldn't initialize a Kubernetes cluster
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init.runWaitControlPlanePhase
        cmd/kubeadm/app/cmd/phases/init/waitcontrolplane.go:108
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
        cmd/kubeadm/app/cmd/phases/workflow/runner.go:234
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
        cmd/kubeadm/app/cmd/phases/workflow/runner.go:421
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
        cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1
        cmd/kubeadm/app/cmd/init.go:154
github.com/spf13/cobra.(*Command).execute
        vendor/github.com/spf13/cobra/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
        vendor/github.com/spf13/cobra/command.go:974
github.com/spf13/cobra.(*Command).Execute
        vendor/github.com/spf13/cobra/command.go:902
k8s.io/kubernetes/cmd/kubeadm/app.Run
        cmd/kubeadm/app/kubeadm.go:50
main.main
        cmd/kubeadm/kubeadm.go:25
runtime.main
        /usr/local/go/src/runtime/proc.go:250
runtime.goexit
        /usr/local/go/src/runtime/asm_arm64.s:1172
error execution phase wait-control-plane
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
        cmd/kubeadm/app/cmd/phases/workflow/runner.go:235
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
        cmd/kubeadm/app/cmd/phases/workflow/runner.go:421
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
        cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1
        cmd/kubeadm/app/cmd/init.go:154
github.com/spf13/cobra.(*Command).execute
        vendor/github.com/spf13/cobra/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
        vendor/github.com/spf13/cobra/command.go:974
github.com/spf13/cobra.(*Command).Execute
        vendor/github.com/spf13/cobra/command.go:902
k8s.io/kubernetes/cmd/kubeadm/app.Run
        cmd/kubeadm/app/kubeadm.go:50
main.main
        cmd/kubeadm/kubeadm.go:25
runtime.main
        /usr/local/go/src/runtime/proc.go:250
runtime.goexit
        /usr/local/go/src/runtime/asm_arm64.s:1172

What you expected to happen:
Cluster to start.

How to reproduce it (as minimally and precisely as possible):
Run kind-with-reg.script
Anything else we need to know?:

Environment:

  • kind version: (use kind version):kind version 0.17.0
  • Runtime info: (use docker info or podman info): Docker version 20.10.22, build 3a2c30b
  • OS (e.g. from /etc/os-release):
  • Kubernetes version: (use kubectl version):
    Client Version: v1.25.4
    Kustomize Version: v4.5.7
  • Any proxies or other special environment settings?: no
@kcirtapfromspace kcirtapfromspace added the kind/bug Categorizes issue or PR as related to a bug. label Feb 3, 2023
@BenTheElder
Copy link
Member

#2875 covers the "no longer reflects guidance for using containerd"

Your attempt didn't work because you need to have the /etc/containerd/certs.d/... directory exist within the kind nodes.

You could use an extraMount to accomplish this.

/kind support
/remove-kind bug

@k8s-ci-robot k8s-ci-robot added kind/support Categorizes issue or PR as a support question. and removed kind/bug Categorizes issue or PR as related to a bug. labels Feb 6, 2023
@BenTheElder BenTheElder self-assigned this May 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@BenTheElder BenTheElder

Labels
kind/support Categorizes issue or PR as a support question.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@BenTheElder @k8s-ci-robot @kcirtapfromspace