gc: unable to remove pod (device/resource busy error)

[ErnestG4]

MOVED FROM https://coreos.zendesk.com/agent/tickets/11546

I’ve also noticed that we can’t correctly GC old pods. I’m not sure if this is something to do with our setup [1]

maybe these? #2878 #2804

or the fact were using systemd templates

switched over to doing this #1171 which eliminates all the subgroup errors, but im still seeing the stage1/rootfs: device or resource busy problem.

Sep 22 14:33:59.625803 rkt: rm: unable to remove pod "1676fb70-7f56-47bc-ae37-cc4afce6a310": remove /mnt/vol1/rkt/pods/exited-garbage/1676fb70-7f56-47bc-ae37-cc4afce6a310/stage1/rootfs: device or resource busy

[1]

$ sudo rkt gc
gc: pod "10491b62-6d44-4564-8aab-2d1fcdcde301" not removed: still within grace period (30m0s)
gc: pod "24cc6a95-9138-4bed-ac1e-cecdb1086e8a" not removed: still within grace period (30m0s)
Garbage collecting pod "9abc813a-8815-409b-8d90-846ae3d14dec"
stage1 gc: error removing subcgroup "/sys/fs/cgroup/cpu,cpuacct/machine.slice/rkt-start-java@5.service/system.slice/centos7-onload-java-operations.service": remove /sys/fs/cgroup/cpu,cpuacct/machine.slice/rkt-start-java@5.service/system.slice/centos7-onload-java-operations.service/cpu.rt_period_us: operation not permitted
stage1 gc: error removing subcgroup "/sys/fs/cgroup/devices/machine.slice/rkt-start-java@5.service/system.slice/centos7-onload-java-operations.service": remove /sys/fs/cgroup/devices/machine.slice/rkt-start-java@5.service/system.slice/centos7-onload-java-operations.service/devices.list: operation not permitted
stage1 gc: error removing subcgroup "/sys/fs/cgroup/systemd/machine.slice/rkt-start-java@5.service/system.slice/centos7-onload-java-operations.service": remove /sys/fs/cgroup/systemd/machine.slice/rkt-start-java@5.service/system.slice/centos7-onload-java-operations.service/cgroup.clone_children: operation not permitted
stage1 gc: error removing subcgroup "/sys/fs/cgroup/cpuacct/machine.slice/rkt-start-java@5.service/system.slice/centos7-onload-java-operations.service": remove /sys/fs/cgroup/cpuacct/machine.slice/rkt-start-java@5.service/system.slice/centos7-onload-java-operations.service/cpu.rt_period_us: operation not permitted
stage1 gc: error removing subcgroup "/sys/fs/cgroup/memory/machine.slice/rkt-start-java@5.service/system.slice/centos7-onload-java-operations.service": remove /sys/fs/cgroup/memory/machine.slice/rkt-start-java@5.service/system.slice/centos7-onload-java-operations.service/memory.kmem.tcp.max_usage_in_bytes: operation not permitted
stage1 gc: error removing subcgroup "/sys/fs/cgroup/cpu/machine.slice/rkt-start-java@5.service/system.slice/centos7-onload-java-operations.service": remove /sys/fs/cgroup/cpu/machine.slice/rkt-start-java@5.service/system.slice/centos7-onload-java-operations.service/cpu.rt_period_us: operation not permitted
stage1 gc: error removing subcgroup "/sys/fs/cgroup/systemd/machine.slice/rkt-start-java@5.service/system.slice/systemd-journald.service": remove /sys/fs/cgroup/systemd/machine.slice/rkt-start-java@5.service/system.slice/systemd-journald.service/cgroup.clone_children: operation not permitted
stage1 gc: error removing subcgroup "/sys/fs/cgroup/cpu,cpuacct/machine.slice/rkt-start-java@5.service/system.slice": remove /sys/fs/cgroup/cpu,cpuacct/machine.slice/rkt-start-java@5.service/system.slice/centos7-onload-java-operations.service/cpu.rt_period_us: operation not permitted
stage1 gc: error removing subcgroup "/sys/fs/cgroup/devices/machine.slice/rkt-start-java@5.service/system.slice": remove /sys/fs/cgroup/devices/machine.slice/rkt-start-java@5.service/system.slice/centos7-onload-java-operations.service/devices.list: operation not permitted
stage1 gc: error removing subcgroup "/sys/fs/cgroup/cpuacct/machine.slice/rkt-start-java@5.service/system.slice": remove /sys/fs/cgroup/cpuacct/machine.slice/rkt-start-java@5.service/system.slice/centos7-onload-java-operations.service/cpu.rt_period_us: operation not permitted
stage1 gc: error removing subcgroup "/sys/fs/cgroup/systemd/machine.slice/rkt-start-java@5.service/system.slice": remove /sys/fs/cgroup/systemd/machine.slice/rkt-start-java@5.service/system.slice/centos7-onload-java-operations.service/cgroup.clone_children: operation not permitted
stage1 gc: error removing subcgroup "/sys/fs/cgroup/memory/machine.slice/rkt-start-java@5.service/system.slice": remove /sys/fs/cgroup/memory/machine.slice/rkt-start-java@5.service/system.slice/centos7-onload-java-operations.service/memory.kmem.tcp.max_usage_in_bytes: operation not permitted
stage1 gc: error removing subcgroup "/sys/fs/cgroup/systemd/machine.slice/rkt-start-java@5.service/init.scope": remove /sys/fs/cgroup/systemd/machine.slice/rkt-start-java@5.service/init.scope/cgroup.clone_children: operation not permitted
stage1 gc: error removing subcgroup "/sys/fs/cgroup/cpu/machine.slice/rkt-start-java@5.service/system.slice": remove /sys/fs/cgroup/cpu/machine.slice/rkt-start-java@5.service/system.slice/centos7-onload-java-operations.service/cpu.rt_period_us: operation not permitted
stage1 gc: error removing subcgroup "/sys/fs/cgroup/cpu,cpuacct/machine.slice/rkt-start-java@5.service": remove /sys/fs/cgroup/cpu,cpuacct/machine.slice/rkt-start-java@5.service/system.slice/centos7-onload-java-operations.service/cpu.rt_period_us: operation not permitted
stage1 gc: error removing subcgroup "/sys/fs/cgroup/cpuacct/machine.slice/rkt-start-java@5.service": remove /sys/fs/cgroup/cpuacct/machine.slice/rkt-start-java@5.service/system.slice/centos7-onload-java-operations.service/cpu.rt_period_us: operation not permitted
stage1 gc: error removing subcgroup "/sys/fs/cgroup/devices/machine.slice/rkt-start-java@5.service": remove /sys/fs/cgroup/devices/machine.slice/rkt-start-java@5.service/system.slice/centos7-onload-java-operations.service/devices.list: operation not permitted
stage1 gc: error removing subcgroup "/sys/fs/cgroup/systemd/machine.slice/rkt-start-java@5.service": remove /sys/fs/cgroup/systemd/machine.slice/rkt-start-java@5.service/system.slice/centos7-onload-java-operations.service/cgroup.clone_children: operation not permitted
stage1 gc: error removing subcgroup "/sys/fs/cgroup/memory/machine.slice/rkt-start-java@5.service": remove /sys/fs/cgroup/memory/machine.slice/rkt-start-java@5.service/system.slice/centos7-onload-java-operations.service/memory.kmem.tcp.max_usage_in_bytes: operation not permitted
stage1 gc: error removing subcgroup "/sys/fs/cgroup/blkio/machine.slice/rkt-start-java@5.service": remove /sys/fs/cgroup/blkio/machine.slice/rkt-start-java@5.service/blkio.io_queued_recursive: operation not permitted
stage1 gc: error removing subcgroup "/sys/fs/cgroup/cpu/machine.slice/rkt-start-java@5.service": remove /sys/fs/cgroup/cpu/machine.slice/rkt-start-java@5.service/system.slice/centos7-onload-java-operations.service/cpu.rt_period_us: operation not permitted
gc: unable to remove pod "9abc813a-8815-409b-8d90-846ae3d14dec": remove /mnt/vol1/rkt/pods/exited-garbage/9abc813a-8815-409b-8d90-846ae3d14dec/stage1/rootfs: device or resource busy

[lucab]

Is this a centos with a pre-4.x kernel by chance? If so, it is known to have a bad behavior due to being an old version with heavy patching, see #1922.

[bcg62]

yes, its centos7 which i believe is 3.10

[pkmiec]

I just upgraded to CentOS 7.4 which comes with kernel 3.10.0-693. I am no longer able to reproduce the device busy problems.