crio on minikube: could not add IP address to "cni0": permission denied #3555

medyagh · 2020-04-10T08:51:03Z

hi we have added crio run-time to our newest minikube drivers (docker and podman). we have a an issue that was hoping you could help us find the root cause,

currently only in docker driver (not our VM drivers) the coreDNS container is stuck in creating and gives this error:

Warning  FailedCreatePodSandBox  5m56s                kubelet, crio-20200409t233749.209311739-13478  Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-5d4dd4b4db-jbm6d_kube-system_77d4783a-a3dd-411e-94f5-7832cc34a3dc_0(9be71bb67410c84168e0a5b940360b84c0a15b1e1acb877d19cec4259cd3f442): failed to set bridge addr: could not add IP address to "cni0": permission denied

so the error is:

failed to set bridge addr: could not add IP address to "cni0": permission denied

but doesnt say what kind of permission it needs and any way we could see more details?

we apply the same cni overlay network for containerd (and containerd works in docker driver) but crio doesnt work. any chance anyone here knows why?

here is the related issue on minikube repo if you need more debugging notes: kubernetes/minikube#7380

The text was updated successfully, but these errors were encountered:

haircommander · 2020-04-13T13:25:54Z

@mccv1r0 anything come to mind?

vikramkhatri · 2020-04-23T15:53:03Z

I am having the same problem but it happens only on one node. I created 10 replicas of a busybox and they got created on other nodes except one node.

  Warning  FailedCreatePodSandBox  0s    kubelet, sc1.xxx.local  Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_busybox-deployment-66458f7d4b-84fjw_default_08b13ded-bcda-4d3e-bd2d-b871eac7fd2f_0(082b0b54aaa9fb86814c1a91d7dd75ef2cef46cfb2f6656cb7c2ee4ef018b190): failed to set bridge addr: could not add IP address to "cni0": permission denied

# kgp -o wide
NAME                                  READY   STATUS              RESTARTS   AGE     IP           NODE            NOMINATED NODE   READINESS GATES
busybox-deployment-66458f7d4b-49jlh   1/1     Running             0          6m29s   10.88.0.20   sc3.xxx.local   <none>           <none>
busybox-deployment-66458f7d4b-4kdms   1/1     Running             0          6m29s   10.88.0.20   sc2.xxx.local   <none>           <none>
busybox-deployment-66458f7d4b-6pt5z   1/1     Running             0          6m29s   10.88.0.19   sc3.xxx.local   <none>           <none>
busybox-deployment-66458f7d4b-84fjw   0/1     ContainerCreating   0          6m29s   <none>       sc1.xxx.local   <none>           <none>
busybox-deployment-66458f7d4b-gbmpv   0/1     ContainerCreating   0          6m29s   <none>       sc1.xxx.local   <none>           <none>
busybox-deployment-66458f7d4b-k9fcq   1/1     Running             0          6m29s   10.88.0.19   sc2.xxx.local   <none>           <none>
busybox-deployment-66458f7d4b-lrfnw   1/1     Running             0          6m29s   10.88.0.17   sc2.xxx.local   <none>           <none>
busybox-deployment-66458f7d4b-p4r5k   1/1     Running             0          6m29s   10.88.0.18   sc2.xxx.local   <none>           <none>
busybox-deployment-66458f7d4b-sn492   0/1     ContainerCreating   0          6m29s   <none>       sc1.xxx.local   <none>           <none>
busybox-deployment-66458f7d4b-w4jz2   1/1     Running             0          6m29s   10.88.0.21   sc3.xxx.local   <none>           <none>

Version info:

# runc --version
runc version 1.0.0-rc10+dev
commit: bf0a8e17471347407fe9e856d4f3ff61beaf2fea
spec: 1.0.2

# kubectl -n kube-system exec -it calicoctl -- calicoctl version
Client Version:    v3.13.3
Git commit:        eb796e31
Cluster Version:   v3.13.2
Cluster Type:      k8s,kdd,bgp,kubeadm

# crio --version
crio version 1.16.6
commit: "5fb673830826e05bb3d325c0b85a62f673ba05d5-dirty"

# kubectl version --short
Client Version: v1.16.9
Server Version: v1.16.9

kungfoome · 2020-05-11T08:48:35Z

Getting the same issue. Restart the server and i am now getting the same error. This is on kubernetes and not minikube.

OS: Fedora 32

runc --version
runc version 1.0.0-rc10+dev
commit: fbdbaf85ecbc0e077f336c03062710435607dbf1
spec: 1.0.1-dev

crio --version
crio version
Version:       1.18.0
GitCommit:     7d79f42b28ad00cf2e7d86604a5a4007303ac328
GitTreeState:  clean
BuildDate:     2020-04-29T15:16:34Z
GoVersion:     go1.14.2
Compiler:      gc

kubectl version --short
Client Version: v1.18.2
Server Version: v1.18.2

Getting this error for coredns and k8s dashboard:

kubectl describe pod coredns-8574cdbd48-5mptz -n kube-system

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-8574cdbd48-5mptz_kube-system_17e7b330-bd19-4278-bdc3-a8fda773c2e4_0(4c572046d936802541db7d1f59f80a71fe128fea006299dad8eb0b618c65aaff): failed to set bridge addr: could not add IP address to "cni0": permission denied

journalctl -xeu crio -o cat

time="2020-05-11 11:46:29.183019215+03:00" level=info msg="Got pod network &{Name:coredns-8574cdbd48-5mptz Namespace:kube-system ID:8d950aefaffae29200c9845e31294a7512d65ff3a0f4bcbfc3b2d42d75649008 NetNS:/proc/84777/ns/net Networks:[] RuntimeConfig:map[crio:{IP: MAC: PortMappings:[] Bandwidth:<nil> IpRanges:[]}]}"
time="2020-05-11 11:46:29.183212847+03:00" level=info msg="About to add CNI network crio (type=bridge)"
time="2020-05-11 11:46:31.227092309+03:00" level=error msg="Error adding network: failed to set bridge addr: could not add IP address to \"cni0\": permission denied"
time="2020-05-11 11:46:31.227236702+03:00" level=error msg="Error while adding pod to CNI network \"crio\": failed to set bridge addr: could not add IP address to \"cni0\": permission denied"
time="2020-05-11 11:46:31.227748839+03:00" level=info msg="Got pod network &{Name:coredns-8574cdbd48-5mptz Namespace:kube-system ID:8d950aefaffae29200c9845e31294a7512d65ff3a0f4bcbfc3b2d42d75649008 NetNS:/proc/84777/ns/net Networks:[] RuntimeConfig:map[crio:{IP: MAC: PortMappings:[] Bandwidth:<nil> IpRanges:[]}]}"
time="2020-05-11 11:46:31.227776470+03:00" level=info msg="About to del CNI network crio (type=bridge)"
time="2020-05-11 11:46:31.260075733+03:00" level=error msg="Error deleting network: running [/usr/sbin/iptables -t nat -D POSTROUTING -s 10.85.4.74 -j CNI-68677a39fdbc25528c895f0a -m comment --comment name: \"crio\" id: \"8d950aefaffae29200c9845e31294a7512d65ff3a0f4bcbfc3b2d42d75649008\" --wait]: exit status 2: iptables v1.8.4 (legacy): Couldn't load target `CNI-68677a39fdbc25528c895f0a':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n"
time="2020-05-11 11:46:31.260236211+03:00" level=error msg="Error while removing pod from CNI network \"crio\": running [/usr/sbin/iptables -t nat -D POSTROUTING -s 10.85.4.74 -j CNI-68677a39fdbc25528c895f0a -m comment --comment name: \"crio\" id: \"8d950aefaffae29200c9845e31294a7512d65ff3a0f4bcbfc3b2d42d75649008\" --wait]: exit status 2: iptables v1.8.4 (legacy): Couldn't load target `CNI-68677a39fdbc25528c895f0a':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n"
time="2020-05-11 11:46:31.260329886+03:00" level=error msg="Error stopping network on cleanup: failed to destroy network for pod sandbox k8s_coredns-8574cdbd48-5mptz_kube-system_17e7b330-bd19-4278-bdc3-a8fda773c2e4_0(8d950aefaffae29200c9845e31294a7512d65ff3a0f4bcbfc3b2d42d75649008): running [/usr/sbin/iptables -t nat -D POSTROUTING -s 10.85.4.74 -j CNI-68677a39fdbc25528c895f0a -m comment --comment name: \"crio\" id: \"8d950aefaffae29200c9845e31294a7512d65ff3a0f4bcbfc3b2d42d75649008\" --wait]: exit status 2: iptables v1.8.4 (legacy): Couldn't load target `CNI-68677a39fdbc25528c895f0a':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\n" id=fa585731-c00a-4a30-bf8a-785b32dda218 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox

haircommander · 2020-05-11T15:17:53Z

@afbjorklund have you encountered this?

afbjorklund · 2020-05-11T15:30:04Z

I think that networking was recently changed to use "kindnet" instead, but I can take another look.

kungfoome · 2020-05-11T15:45:16Z

I was able to resolve my issue finally. I don't remember exactly what I did before it started to not work, but to resolve it I disabled ipv6. Once I disabled ipv6 it still gave the same error and then once I updated the bridge config to remove ipv6 range, it started to work ok and the error message went away.

afbjorklund · 2020-05-11T15:54:54Z

No it wasn't kindnet but the "default networking" (mybridge). And it seemed to work better ?

kubernetes/minikube@7aad057

https://github.com/kubernetes/minikube/blob/master/pkg/minikube/bootstrapper/kubeadm/default_cni.go#L25_L45

Reverting that change and going with crio-bridge (instead of rkt.kubernetes.io), to try it again.

afbjorklund · 2020-05-11T16:32:26Z

I wonder if this is related: f4214be (both crio and podman using 10.88.0.0)

100-crio-bridge.conf:            [{ "subnet": "10.88.0.0/16" }],
100-crio-bridge.conf:            [{ "subnet": "1100:200::/24" }]
87-podman-bridge.conflist:              "subnet": "10.88.0.0/16",
k8s.conf:    "subnet": "10.1.0.0/16",

cri-o-1.17: /etc/cni/net.d/100-crio-bridge.conf
podman: /etc/cni/net.d/87-podman-bridge.conflist

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-===================================================================
ii  cri-o-1.17     1.17.3~3     amd64        OCI-based implementation of Kubernetes Container Runtime Interface.
ii  podman         1.9.0~2      amd64        Manage pods, containers and container images.

vivekanandpoojari · 2020-05-23T14:07:51Z

I was able to resolve my issue finally. I don't remember exactly what I did before it started to not work, but to resolve it I disabled ipv6. Once I disabled ipv6 it still gave the same error and then once I updated the bridge config to remove ipv6 range, it started to work ok and the error message went away.

I tried this method and it works. The crio cni files have ipv6 by default. My host did not have IPv6 enabled. This work around is useful in cases IPv6 is disabled on the host or not correctly configured.

@medyagh FYI

haircommander · 2020-05-26T14:56:51Z

Given the work around, can this issue be considered closed, or is there a change folks would like to see?

haircommander · 2020-06-01T18:57:50Z

closing, please reopen if you disagree or the work around does not work for you

yatakoi · 2021-12-12T03:32:15Z

I was able to resolve my issue finally. I don't remember exactly what I did before it started to not work, but to resolve it I disabled ipv6. Once I disabled ipv6 it still gave the same error and then once I updated the bridge config to remove ipv6 range, it started to work ok and the error message went away.

I confirm that enabling IPv6 solves the problem! Thank you!

haircommander · 2021-12-13T17:22:54Z

FYI for folks finding this, we have an alternative CNI file and description of when to use it here

Wang-Kai · 2022-02-02T06:01:49Z

I was able to resolve my issue finally. I don't remember exactly what I did before it started to not work, but to resolve it I disabled ipv6. Once I disabled ipv6 it still gave the same error and then once I updated the bridge config to remove ipv6 range, it started to work ok and the error message went away.

Thx, i fix it with:

sysctl -w net.ipv6.conf.all.disable_ipv6=0
sysctl -w net.ipv6.conf.default.disable_ipv6=0
sysctl -w net.ipv6.conf.tun0.disable_ipv6=0
sysctl -p

anrs · 2022-08-05T01:30:25Z

I was able to resolve my issue finally. I don't remember exactly what I did before it started to not work, but to resolve it I disabled ipv6. Once I disabled ipv6 it still gave the same error and then once I updated the bridge config to remove ipv6 range, it started to work ok and the error message went away.

Thx, i fix it with:
sysctl -w net.ipv6.conf.all.disable_ipv6=0
sysctl -w net.ipv6.conf.default.disable_ipv6=0
sysctl -w net.ipv6.conf.tun0.disable_ipv6=0
sysctl -p

I've encountered the 'permission denied' error before and fixed as @Wang-Kai solution

Jeansen · 2023-12-28T16:21:26Z

I think, this is a misunderstanding. Things like net.ipv6.conf.all.disable_ipv6=0 ENABLE ipv6. As it reads, it says "don't disable ipv6". I just came across this because I had explicitly set net.ipv6.conf.all.disable_ipv6=1 for some configs which in turn made the OP's problem pop up for CoreDNS Pods in my cluster.

medyagh changed the title ~~crio on docker driver in minikube: could not add IP address to "cni0": permission denied~~ crio on minikube: could not add IP address to "cni0": permission denied Apr 10, 2020

haircommander closed this as completed Jun 1, 2020

johnmuth81 mentioned this issue Nov 11, 2021

'minikube start --container-runtime=cri-o --wait=all' fails: time out waiting for condition - v1.24.0 kubernetes/minikube#12928

Closed

utkarsh-singh1 mentioned this issue Feb 1, 2024

Error adding pod to CNI network "crio": plugin type="bridge" failed (add): failed to set bridge addr: could not add IP address to "cni0": permission denied kubernetes/kubernetes#121102

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

crio on minikube: could not add IP address to "cni0": permission denied #3555

crio on minikube: could not add IP address to "cni0": permission denied #3555

medyagh commented Apr 10, 2020 •

edited

Loading

haircommander commented Apr 13, 2020

vikramkhatri commented Apr 23, 2020 •

edited

Loading

kungfoome commented May 11, 2020 •

edited

Loading

haircommander commented May 11, 2020

afbjorklund commented May 11, 2020

kungfoome commented May 11, 2020

afbjorklund commented May 11, 2020

afbjorklund commented May 11, 2020 •

edited

Loading

vivekanandpoojari commented May 23, 2020 •

edited

Loading

haircommander commented May 26, 2020

haircommander commented Jun 1, 2020

yatakoi commented Dec 12, 2021

haircommander commented Dec 13, 2021

Wang-Kai commented Feb 2, 2022

anrs commented Aug 5, 2022

Jeansen commented Dec 28, 2023

crio on minikube: could not add IP address to "cni0": permission denied #3555

crio on minikube: could not add IP address to "cni0": permission denied #3555

Comments

medyagh commented Apr 10, 2020 • edited Loading

haircommander commented Apr 13, 2020

vikramkhatri commented Apr 23, 2020 • edited Loading

kungfoome commented May 11, 2020 • edited Loading

haircommander commented May 11, 2020

afbjorklund commented May 11, 2020

kungfoome commented May 11, 2020

afbjorklund commented May 11, 2020

afbjorklund commented May 11, 2020 • edited Loading

vivekanandpoojari commented May 23, 2020 • edited Loading

haircommander commented May 26, 2020

haircommander commented Jun 1, 2020

yatakoi commented Dec 12, 2021

haircommander commented Dec 13, 2021

Wang-Kai commented Feb 2, 2022

anrs commented Aug 5, 2022

Jeansen commented Dec 28, 2023

medyagh commented Apr 10, 2020 •

edited

Loading

vikramkhatri commented Apr 23, 2020 •

edited

Loading

kungfoome commented May 11, 2020 •

edited

Loading

afbjorklund commented May 11, 2020 •

edited

Loading

vivekanandpoojari commented May 23, 2020 •

edited

Loading