crio failed on master on jenkins: ❌ Exiting due to RUNTIME_ENABLE: #10520

medyagh · 2021-02-19T23:24:47Z

jenkins@debian-jenkins-agent-9:~$ ./minikube-linux-amd64 start --container-runtime=cri-o
😄  minikube v1.17.1 on Debian 9.13 (kvm/amd64)
✨  Automatically selected the docker driver. Other choices: kvm2, ssh
👍  Starting control plane node minikube in cluster minikube
💾  Downloading Kubernetes v1.20.2 preload ...
    > preloaded-images-k8s-v8-v1....: 556.07 MiB / 556.07 MiB  100.00% 186.04 M
🔥  Creating docker container (CPUs=2, Memory=7500MB) ...
❗  Cgroup v2 does not allow setting memory, if you want to set memory, please modify your Grub as instructed in https://docs.docker.com/engine/install/linux-postinstall/#your-k
ernel-does-not-support-cgroup-swap-limit-capabilities
❌  Exiting due to RUNTIME_ENABLE: sudo systemctl start crio: Process exited with status 1
stdout:
stderr:
Job for crio.service failed because the control process exited with error code.
See "systemctl status crio.service" and "journalctl -xe" for details.
😿  If the above advice does not help, please let us know: 
👉  https://github.com/kubernetes/minikube/issues/new/choose

jenkins@debian-jenkins-agent-9:~$ ./minikube-linux-amd64 ssh
docker@minikube:~$ sudo systemctl start crio
Job for crio.service failed because the control process exited with error code.
See "systemctl status crio.service" and "journalctl -xe" for details.
docker@minikube:~$ sudo systemctl status    
● minikube
    State: degraded
     Jobs: 0 queued
   Failed: 2 units
    Since: Fri 2021-02-19 23:21:42 UTC; 2min 6s ago
   CGroup: /docker/c71e9764f24a0360b494d5c39feb92ae47c37b35a18fe8c1c86e077bfeecf6fa
           ├─init.scope 
           │ └─1 /sbin/init
           └─system.slice 
             ├─dbus.service 
             │ └─177 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
             ├─ssh.service 
             │ ├─190 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
             │ ├─469 sshd: docker [priv]
             │ ├─471 sshd: docker@pts/1
             │ ├─472 -bash
             │ ├─504 sudo systemctl status
             │ ├─505 systemctl status
             │ └─506 pager
             └─systemd-journald.service 
               └─166 /lib/systemd/systemd-journald

The text was updated successfully, but these errors were encountered:

medyagh · 2021-02-19T23:31:15Z

jenkins@debian-jenkins-agent-9:~$ ./minikube-linux-amd64 ssh
docker@minikube:~$ sudo systemctl start crio
Job for crio.service failed because the control process exited with error code.
See "systemctl status crio.service" and "journalctl -xe" for details.
docker@minikube:~$ sudo systemctl status    
● minikube
    State: degraded
     Jobs: 0 queued
   Failed: 2 units
    Since: Fri 2021-02-19 23:21:42 UTC; 2min 6s ago
   CGroup: /docker/c71e9764f24a0360b494d5c39feb92ae47c37b35a18fe8c1c86e077bfeecf6fa
           ├─init.scope 
           │ └─1 /sbin/init
           └─system.slice 
             ├─dbus.service 
             │ └─177 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
             ├─ssh.service 
             │ ├─190 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
             │ ├─469 sshd: docker [priv]
             │ ├─471 sshd: docker@pts/1
             │ ├─472 -bash
             │ ├─504 sudo systemctl status
             │ ├─505 systemctl status
             │ └─506 pager
             └─systemd-journald.service 
               └─166 /lib/systemd/systemd-journald
docker@minikube:~$ sudo systemctl restart crio
Job for crio.service failed because the control process exited with error code.
See "systemctl status crio.service" and "journalctl -xe" for details.
docker@minikube:~$ journalctl -xe
Hint: You are currently not seeing messages from other users and the system.
      Users in the 'systemd-journal' group can see all messages. Pass -q to
      turn off this notice.
No journal files were opened due to insufficient permissions.
docker@minikube:~$ sudo journalctl -xe
-- Logs begin at Fri 2021-02-19 23:21:42 UTC, end at Fri 2021-02-19 23:30:22 UTC. --
Feb 19 23:21:42 minikube systemd-journald[166]: Journal started
-- Subject: The journal has been started
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The system journal process has started up, opened the journal
-- files for writing and is now ready to process requests.
Feb 19 23:21:42 minikube systemd-journald[166]: Runtime Journal (/run/log/journal/0059402bd13f40d393fcde71682d2cd1) is 8.0M, max 1.4G, 1.4G free.
-- Subject: Disk space used by the journal
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Runtime Journal (/run/log/journal/0059402bd13f40d393fcde71682d2cd1) is currently using 8.0M.
-- Maximum allowed usage is set to 1.4G.
-- Leaving at least 754.0M free (of currently available 14.7G of disk space).

medyagh · 2021-02-19T23:33:38Z

tstromberg · 2021-02-19T23:39:16Z

Oddly enough, I just tried CRIO with the docker driver on gLinux (rodete) and crio seems to work fine at master there.

● crio.service - Container Runtime Interface for OCI (CRI-O)
     Loaded: loaded (/lib/systemd/system/crio.service; disabled; vendor preset: enabled)
     Active: active (running) since Fri 2021-02-19 23:36:02 UTC; 4min 1s ago
       Docs: https://github.com/cri-o/cri-o
   Main PID: 2193 (crio)
      Tasks: 14
     Memory: 30.8M
     CGroup: /docker/9e7a9bc37e3b6d1eb7a04f56de4b7bbed7d84e2070ae3ccf7f9fd6b2370a476a/system.slice/crio.service
             └─2193 /usr/bin/crio

medyagh · 2021-02-19T23:40:37Z

root@minikube:/home/docker# systemctl status  cri-o
● crio.service - Container Runtime Interface for OCI (CRI-O)
     Loaded: loaded (/lib/systemd/system/crio.service; disabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2021-02-19 23:39:51 UTC; 8s ago
       Docs: https://github.com/cri-o/cri-o
    Process: 2298 ExecStart=/usr/bin/crio $CRIO_CONFIG_OPTIONS $CRIO_RUNTIME_OPTIONS $CRIO_STORAGE_OPTIONS $CRIO_NETWORK_OPTIONS $CRIO_METRICS_OPTIONS (code=exited, stat
us=1/FAILURE)
   Main PID: 2298 (code=exited, status=1/FAILURE)
Feb 19 23:39:51 minikube crio[2298]: time="2021-02-19 23:39:51.958947531Z" level=info msg="Node configuration value for hugetlb cgroup is false"
Feb 19 23:39:51 minikube crio[2298]: time="2021-02-19 23:39:51.958985109Z" level=info msg="Node configuration value for pid cgroup is true"
Feb 19 23:39:51 minikube crio[2298]: time="2021-02-19 23:39:51.959048494Z" level=error msg="Node configuration validation for memoryswap cgroup failed: node not configured with 
memory swap"
Feb 19 23:39:51 minikube crio[2298]: time="2021-02-19 23:39:51.959057754Z" level=info msg="Node configuration value for memoryswap cgroup is false"
Feb 19 23:39:51 minikube crio[2298]: time="2021-02-19 23:39:51.961369440Z" level=info msg="Node configuration value for systemd CollectMode is true"
Feb 19 23:39:51 minikube crio[2298]: time="2021-02-19 23:39:51.973258879Z" level=info msg="Overlay test mount did not indicate whether or not metacopy is being used: failed to m
ount overlay for metacopy check with \"nodev,metacopy=on\" options: invalid argument"
Feb 19 23:39:51 minikube crio[2298]: time="2021-02-19 23:39:51.973380673Z" level=fatal msg="Validating root config: failed to get store to set defaults: failed to mount overlay 
for metacopy check with \"nodev,metacopy=on\" options: invalid argument"

tstromberg · 2021-02-19T23:41:14Z

Seeing the output of sudo journalctl -u crio would be helpful to debug this

medyagh · 2021-02-19T23:49:58Z

@tstromberg said


My gut instinct says that metacopy is likely too new for that kernel, and we might need to tell crio to turn the metacopy feature off. Doing some Googling, it seems like crio is supposed to detect it as an overlayfs capability?  

https://github.com/cri-o/cri-o/issues/3631 seems to reflect this

That issue seems to reference a storage.conf file where metacopy=on might be set?

medyagh · 2021-02-19T23:50:26Z

docker@minikube:~$ sudo journalctl -u crio
-- Logs begin at Fri 2021-02-19 23:43:27 UTC, end at Fri 2021-02-19 23:50:11 UTC. --
Feb 19 23:43:33 minikube systemd[1]: Starting Container Runtime Interface for OCI (CRI-O)...
Feb 19 23:43:33 minikube crio[429]: time="2021-02-19 23:43:33.898017050Z" level=info msg="Node configuration value for hugetlb cgroup is false"
Feb 19 23:43:33 minikube crio[429]: time="2021-02-19 23:43:33.898092965Z" level=info msg="Node configuration value for pid cgroup is true"
Feb 19 23:43:33 minikube crio[429]: time="2021-02-19 23:43:33.898159457Z" level=error msg="Node configuration validation for memoryswap cgroup failed: node not configured with memory swap"
Feb 19 23:43:33 minikube crio[429]: time="2021-02-19 23:43:33.898170650Z" level=info msg="Node configuration value for memoryswap cgroup is false"
Feb 19 23:43:33 minikube crio[429]: time="2021-02-19 23:43:33.900478011Z" level=info msg="Node configuration value for systemd CollectMode is true"
Feb 19 23:43:34 minikube crio[429]: time="2021-02-19 23:43:34.008556444Z" level=warning msg="Overlay test mount did not indicate whether or not metacopy is being used: failed to mount overlay fo
r metacopy check with \"nodev,metacopy=on\" options: invalid argument"
Feb 19 23:43:34 minikube crio[429]: time="2021-02-19 23:43:34.008652265Z" level=fatal msg="Validating root config: failed to get store to set defaults: failed to mount overlay for metacopy check
 with \"nodev,metacopy=on\" options: invalid argument"
Feb 19 23:43:34 minikube systemd[1]: crio.service: Main process exited, code=exited, status=1/FAILURE
Feb 19 23:43:34 minikube systemd[1]: crio.service: Failed with result 'exit-code'.
Feb 19 23:43:34 minikube systemd[1]: Failed to start Container Runtime Interface for OCI (CRI-O).

medyagh · 2021-02-19T23:55:49Z

I confirm

removing metacopy=on from

mountopt = "nodev,metacopy=on"

in
/etc/containers/storage.conf

fixed it



docker@minikube:~$ sudo systemctl restart crio
docker@minikube:~$ sudo systemctl status crio
● crio.service - Container Runtime Interface for OCI (CRI-O)
     Loaded: loaded (/lib/systemd/system/crio.service; disabled; vendor preset: enabled)
     Active: active (running) since Fri 2021-02-19 23:53:42 UTC; 5s ago
       Docs: https://github.com/cri-o/cri-o
   Main PID: 1595 (crio)
      Tasks: 13
     Memory: 59.1M
     CGroup: /docker/a49af12b49486ad7098f56605eea040cf1acdcdabbfb39bea02feaf8ed003448/system.slice/crio.service
             └─1595 /usr/bin/crio
Feb 19 23:53:42 minikube crio[1595]: time="2021-02-19 23:53:42.381416685Z" level=info msg="Node configuration value for systemd CollectMode is true"
Feb 19 23:53:42 minikube crio[1595]: time="2021-02-19 23:53:42.457336357Z" level=info msg="Using default capabilities: CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FSETID, CAP_FOWNER, CAP_SETGID, CAP_SETUID
, CAP_SETPCAP, CAP_NET_BIND_SERVICE, CAP_KILL"
Feb 19 23:53:42 minikube crio[1595]: time="2021-02-19 23:53:42.474676548Z" level=info msg="Conmon does support the --sync option"
Feb 19 23:53:42 minikube crio[1595]: time="2021-02-19 23:53:42.474915381Z" level=info msg="No seccomp profile specified, using the internal default"
Feb 19 23:53:42 minikube crio[1595]: time="2021-02-19 23:53:42.474937974Z" level=info msg="AppArmor is disabled by the system or at CRI-O build-time"
Feb 19 23:53:42 minikube crio[1595]: time="2021-02-19 23:53:42.508606221Z" level=info msg="Found CNI network crio (type=bridge) at /etc/cni/net.d/100-crio-bridge.conf"
Feb 19 23:53:42 minikube crio[1595]: time="2021-02-19 23:53:42.629765486Z" level=info msg="Found CNI network 200-loopback.conf (type=loopback) at /etc/cni/net.d/200-loopback.conf"
Feb 19 23:53:42 minikube crio[1595]: time="2021-02-19 23:53:42.702783184Z" level=info msg="Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist"
Feb 19 23:53:42 minikube crio[1595]: time="2021-02-19 23:53:42.702834050Z" level=info msg="Update default CNI network name to crio"
Feb 19 23:53:42 minikube systemd[1]: Started Container Runtime Interface for OCI (CRI-O).

medyagh · 2021-02-19T23:59:54Z

currently for VM drivers we have a config template for /etc/containers/storage.conf https://github.com/medyagh/minikube/blob/3d5809b98846da4929e1256141ea051f0a67f6d5/deploy/iso/minikube-iso/package/crio-bin/crio.conf.default#L32

but for KIC drivers (docker,podman) and crio runtime we dont hhave a config
I would accept a PR that will generate the /etc/containers/storage.conf

afbjorklund · 2021-02-20T08:16:28Z

This is known upstream: cri-o/cri-o#4574

we should really fix this in the package itself, it isn't too hard to auto-detect, and not worth end-user time/energy :)

afbjorklund · 2021-02-20T08:18:21Z

@tstromberg : Basically they assume that Ubuntu 20.04 is running with the Ubuntu 20.04 kernel, not as a fakenode

sharifelgamal self-assigned this Feb 20, 2021

sharifelgamal added this to the v1.18.0 milestone Feb 20, 2021

sharifelgamal mentioned this issue Feb 20, 2021

fix crio by hardcoding mountopt #10528

Merged

medyagh closed this as completed in #10528 Feb 20, 2021

afbjorklund added co/runtime/crio CRIO related issues co/docker-driver Issues related to kubernetes in container kind/bug Categorizes issue or PR as related to a bug. labels Feb 20, 2021

afbjorklund mentioned this issue Feb 20, 2021

Upgrade CRI-O to 1.20.0 #10478

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

crio failed on master on jenkins: ❌ Exiting due to RUNTIME_ENABLE: #10520

crio failed on master on jenkins: ❌ Exiting due to RUNTIME_ENABLE: #10520

medyagh commented Feb 19, 2021 •

edited

Loading

medyagh commented Feb 19, 2021 •

edited

Loading

medyagh commented Feb 19, 2021 •

edited

Loading

tstromberg commented Feb 19, 2021 •

edited

Loading

medyagh commented Feb 19, 2021

tstromberg commented Feb 19, 2021 •

edited

Loading

medyagh commented Feb 19, 2021

medyagh commented Feb 19, 2021

medyagh commented Feb 19, 2021

medyagh commented Feb 19, 2021

afbjorklund commented Feb 20, 2021 •

edited

Loading

afbjorklund commented Feb 20, 2021 •

edited

Loading

crio failed on master on jenkins: ❌ Exiting due to RUNTIME_ENABLE: #10520

crio failed on master on jenkins: ❌ Exiting due to RUNTIME_ENABLE: #10520

Comments

medyagh commented Feb 19, 2021 • edited Loading

medyagh commented Feb 19, 2021 • edited Loading

medyagh commented Feb 19, 2021 • edited Loading

tstromberg commented Feb 19, 2021 • edited Loading

medyagh commented Feb 19, 2021

tstromberg commented Feb 19, 2021 • edited Loading

medyagh commented Feb 19, 2021

medyagh commented Feb 19, 2021

medyagh commented Feb 19, 2021

medyagh commented Feb 19, 2021

afbjorklund commented Feb 20, 2021 • edited Loading

afbjorklund commented Feb 20, 2021 • edited Loading

medyagh commented Feb 19, 2021 •

edited

Loading

medyagh commented Feb 19, 2021 •

edited

Loading

medyagh commented Feb 19, 2021 •

edited

Loading

tstromberg commented Feb 19, 2021 •

edited

Loading

tstromberg commented Feb 19, 2021 •

edited

Loading

afbjorklund commented Feb 20, 2021 •

edited

Loading

afbjorklund commented Feb 20, 2021 •

edited

Loading