rm: cannot remove './fsroot/var/lib/docker': Device or resource busy

[stcheng]

[ FAIL LOG START ] [ target/sonic-vs.img.gz ]
+ DOCKER_VERSION=5:18.09.2~3-0~debian-stretch
+ LINUX_KERNEL_VERSION=4.9.0-8-2
+ FILESYSTEM_ROOT=./fsroot
+ PLATFORM_DIR=platform
+ HOSTNAME=sonic
+ DEFAULT_USERINFO='Default **** user,,,'
+ . ./onie-image.conf
++ : 32768
++ : x86_64
++ : generic
++ ONIEIMAGE_VERSION=r0
++ FILESYSTEM_SQUASHFS=fs.squashfs
++ ONIE_INSTALLER_PAYLOAD=fs.zip
++ FILESYSTEM_DOCKERFS=dockerfs.tar.gz
++ DOCKERFS_DIR=docker
++ OUTPUT_ONIE_IMAGE=target/sonic-generic.bin
++ OUTPUT_RAW_IMAGE=target/sonic-generic.raw
++ RAW_IMAGE_DISK_SIZE=1024
++ OUTPUT_KVM_IMAGE=target/sonic-generic.img
++ KVM_IMAGE_DISK_SIZE=16
++ OUTPUT_ABOOT_IMAGE=target/sonic-aboot-generic.swi
++ ABOOT_BOOT_IMAGE=.sonic-boot.swi
+ '[' -n 32768 ']'
+ '[' -n fs.zip ']'
+ '[' -n fs.squashfs ']'
+ [[ -d ./fsroot ]]
+ sudo rm -rf ./fsroot
rm: cannot remove './fsroot/var/lib/docker': Device or resource busy
+ die 'Failed to clean chroot directory'
+ local 'message=Failed to clean chroot directory'
+ warn 'Failed to clean chroot directory'
+ local 'message=Failed to clean chroot directory'
+ echo 'Failed to clean chroot directory'
Failed to clean chroot directory
+ exit 1

[stcheng]

i'll check again on the reason.

[baxia-lan]

Same issue here, any update?

[billie-alsup]

I also received this error today while trying to build master branch for PLATFORM=broadcom. A google search seems to show this is a known problem for older kernels, where docker filesystem mounts can leak into other namespaces (moby/moby#34198). That URL in particular deals with cAdvisor, but I thought it might apply in general to docker usage. I had another branch (different workspace obviously) building in parallel. It completed successfully, so I wondered if that build's mount was what had "leaked". When I rebuilt the failing image, without a second simultaneous build occurring, I was able to successfully complete the build. So I can't definitively say that it was related to this reported "leakage" of mounts into other namespaces, but I can say that it built successfully in isolation on a 2nd attempt.

[inspurSDN]

I have the same issue on master, any solution?

[timmy00274672]

@inspurSDN

Have you manually deleted this folder and remaked again?

[inspurSDN]

@timmy00274672 remake is work, thanks.

[armartirosyan]

The problem is still present in the master branch?
Any ideas why this is happening?

[MiaoXuHui]

Maybe it is a code bug, not a docker issue, we need to umount fsroot/var/lib/docker and fsroot, then the fsroot can be removed.

133e8fcc4ec4:/sonic$ sudo rm -rvf fsroot/
rm: cannot remove 'fsroot/var/lib/docker': Device or resource busy

133e8fcc4ec4:/sonic$ mount | grep fsroot
/dev/sdc on /sonic/fsroot type ext4 (rw,relatime)
/dev/sdc on /sonic/fsroot/var/lib/docker type ext4 (rw,relatime)

133e8fcc4ec4:/sonic$ sudo umount /sonic/fsroot/var/lib/docker
133e8fcc4ec4:/sonic$ sudo rm -rf fsroot/var/lib/docker
133e8fcc4ec4:/sonic$ sudo umount /sonic/fsroot
133e8fcc4ec4:/sonic$ sudo rm -r fsroot

The problem is still present in the master branch?
Any ideas why this is happening?