--image-repository + coredns pulling issue due to namespace removed

[Hugo54]

What Happened?

Hi,

When I set the image-repostitory, after pulling a bunch of image, the process crash on the coredns image. After activate the logging it seems a bug during string append or something like that.

The correct repository image name is grc.io/coredns/coredns:v1.8.4 but when you set your repostory it become x.x.x.x/coredns:v1.8.4. Then the image is not found.

[user@localhost root]$ minikube version
minikube version: v1.24.0
commit: 76b94fb3c4e8ac5062daf70d60cf03ddcc0a741b

Attach the log file

[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'

stderr:
[WARNING FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/3.10.0-1160.49.1.el7.x86_64\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image 10.0.2.15/coredns:v1.8.4: output: Error response from daemon: manifest for 10.0.2.15/coredns:v1.8.4 not found: manifest unknown: manifest unknown
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...
To see the stack trace of this error execute with --v=5 or higher

Operating System

Redhat/Fedora

Driver

Docker

[afbjorklund]

The coredns upgrade was a disaster. Not sure if the upstream "special case" should be kept or ignored.

• Repository and tag for CoreDNS changed in 1.8.0 (Kubernetes 1.21) #11424

There seemed to be a PR for it, but not sure if a fix ever got merged ?

• fix coredns image name with custom image repository #12411

---

It should probably just sed out the GCR (k8s.gcr.io/), and keep whatever random prefix that the image has.

Not sure if there was ever an explanation for the subdirectory, maybe an accident pulling from docker.io ?
But in that case, where did the v come from ? Seems to be rather arbitrary, but I guess we live with it now...

docker.io/coredns/coredns:1.7.0 -> k8s.gcr.io/coredns:1.7.0

docker.io/coredns/coredns:1.8.6 -> k8s.gcr.io/coredns/coredns:v1.8.6

[Hugo54]

Thanks for you reply. I will double check if k8s.gcr.io was not removed from our virtual registry.

After looking at all the log. Because the option of insecure registry was proposed, I supposed that I can use an http repository. However I find a lot of errors related to https. I tried to configure a self signed certificate but it seems to be KO. Can we really use a private registry without a certificate ?

Maybe this issue trigger the fallback on docker.io ? with the wrong namespace.

Regards

I0111 08:39:00.097257 8111 command_runner.go:127] ! curl: (60) SSL certificate problem: self signed certificate
I0111 08:39:00.097282 8111 command_runner.go:127] ! More details here: https://curl.haxx.se/docs/sslcerts.html
I0111 08:39:00.097290 8111 command_runner.go:127] ! curl failed to verify the legitimacy of the server and therefore could not
I0111 08:39:00.097301 8111 command_runner.go:127] ! establish a secure connection to it. To learn more about this situation and
I0111 08:39:00.097307 8111 command_runner.go:127] ! how to fix it, please visit the web page mentioned above.
W0111 08:39:00.097340 8111 start.go:664] [curl -sS -m 2 https://10.0.2.15/] failed: curl -sS -m 2 https://10.0.2.15/: Process exited with status 60
stdout:

stderr:
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
I0111 08:39:00.097355 8111 command_runner.go:127] > systemd 245 (245.4-4ubuntu3.13)
I0111 08:39:00.097378 8111 command_runner.go:127] > +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid
W0111 08:39:00.097455 8111 out.go:241] ! This container is having trouble accessing https://10.0.2.15
! This container is having trouble accessing https://10.0.2.15

[Hugo54]

Forget my last message, httpS doesn't seems a blocker. Just to let you know, I made a dirty hack as workaround with a reverse proxy with some url rewriting and it works. So the main issue/bug is definitely this repository url for coredns.

Thanks,

[afbjorklund]

Eventually might have to provide a place to put custom SSL certs, including corporate SSL proxies that steams all your mail.

SSL_CERT_FILE
SSL_CERT_DIR

[phvogtrbi]

I have the same problem, but unfortunately in my company setting, I cannot easily use the hack, that @Hugo54 did.
Is there a way to do this correctly?

[Hugo54]

@phvogtrbi I had also to build a local docker repostitory to be connected to our compagny repository (with auth & https). And then I used it on my minikube with docker engine configured on it AND the proxy for the domain hack. But I finaly got to many problems so I surrended. I use now services on cloud ...

[phvogtrbi]

I pulled + tagged + pushed all necessary docker images on my own repo and used this in --image-repository. This includes tagging the coredns/coredns also as only coredns. This way minikube starts.

The only drawback is: I cannot change this repo later, so for example to install an addon, I have to do the same pull + tag + push again.
If someone knows a way to change the --image-repository once minikube is set up, please let me know. Thx.

[viluon]

I believe this issue was incorrectly closed despite the bug never having been fixed.