LXD drivers fail to shutdown privileged container if system datastore is symlinked

[dann1]

Description
When having symlinked datastores, ex. datastores directories linked to an NFS mountpoint

root@ubuntu1804-lxd-nfs-a8194-2:~# cat /etc/fstab 
/dev/vda1 / ext4 rw,relatime,nobarrier,commit=600 0 0
LABEL=UEFI	/boot/efi	vfat	defaults	0 0
192.168.150.1:/srv /srv nfs defaults 0 0
root@ubuntu1804-lxd-nfs-a8194-2:~# ls -lh /var/lib/one/datastores/
total 0
lrwxrwxrwx 1 oneadmin oneadmin 6 Apr  4 05:55 0 -> /srv/0
lrwxrwxrwx 1 oneadmin oneadmin 6 Apr  4 05:55 1 -> /srv/1

LXD drivers look for the real path of a mountpoint in order to perform an unmap action. However, if the container is privileged, it will fail because the container directory won't allow oneadmin to read inside them because of the permission bits

root@ubuntu1804-lxd-nfs-a8194-2:/var/lib/lxd/storage-pools/default/containers# ll
total 44
drwx--x--x 11 root   root   4096 Apr  5 02:44 ./
drwx--x--x  3 root   root   4096 Apr  4 06:02 ../
drwx------  3 root   root   4096 Apr  4 06:02 one-0/
drwx--x--x  3 100000 100000 4096 Apr  5 02:43 one-10/
drwx------  3 root   root   4096 Apr  5 02:44 one-11/
drwx------  3 root   root   4096 Apr  4 07:10 one-3/
drwx------  3 root   root   4096 Apr  5 02:30 one-4/
drwx------  3 root   root   4096 Apr  5 02:33 one-6/
drwx--x--x  3 100000 100000 4096 Apr  5 02:33 one-7/
drwx------  3 root   root   4096 Apr  5 02:40 one-8/
drwx------  3 root   root   4096 Apr  5 02:43 one-9/ 

In this case only one-7 and one-10 are unprivileged, the rest of the containers will fail to power off giving this error in the VM log

Fri Apr 5 02:45:03 2019 [Z0][VM][I]: New LCM state is SHUTDOWN_POWEROFF
Fri Apr 5 02:45:07 2019 [Z0][VMM][I]: Command execution fail: cat << EOT | /var/tmp/one/vmm/lxd/shutdown 'one-11' 'ubuntu1804-lxd-nfs-a8194-2.test' 11 ubuntu1804-lxd-nfs-a8194-2.test
Fri Apr 5 02:45:07 2019 [Z0][VMM][I]: shutdown: Using qcow2 mapper for /var/lib/one/datastores/0/11/disk.0
Fri Apr 5 02:45:07 2019 [Z0][VMM][I]: shutdown: Unmapping disk at /var/lib/lxd/storage-pools/default/containers/one-11/rootfs
Fri Apr 5 02:45:07 2019 [Z0][VMM][I]: /var/tmp/one/vmm/lxd/mapper.rb:164:in `realpath': Permission denied @ realpath_rec - /var/lib/lxd/storage-pools/default/containers/one-11/rootfs (Errno::EACCES)
Fri Apr 5 02:45:07 2019 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/mapper.rb:164:in `unmap'
Fri Apr 5 02:45:07 2019 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/container.rb:376:in `public_send'
Fri Apr 5 02:45:07 2019 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/container.rb:376:in `setup_disk'
Fri Apr 5 02:45:07 2019 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/container.rb:258:in `block in setup_storage'
Fri Apr 5 02:45:07 2019 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/container.rb:251:in `each'
Fri Apr 5 02:45:07 2019 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/container.rb:251:in `setup_storage'
Fri Apr 5 02:45:07 2019 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/shutdown:56:in `<main>'
Fri Apr 5 02:45:07 2019 [Z0][VMM][I]: ExitCode: 1
Fri Apr 5 02:45:07 2019 [Z0][VMM][I]: Failed to execute virtualization driver operation: shutdown.
Fri Apr 5 02:45:07 2019 [Z0][VMM][E]: Error shutting down VM
Fri Apr 5 02:45:07 2019 [Z0][VM][I]: New LCM state is RUNNING
Fri Apr 5 02:45:07 2019 [Z0][LCM][I]: Fail to shutdown VM. Assuming that the VM is still RUNNING (will poll VM).

To Reproduce

• Setup a symlinked system datastore
• Deploy a privileged container (LXD_SECURITY_PRIVILEGED = true) in the symlinked datastore
• Attempt Shutdown, and it will fail

Expected behavior
If the mountpoint whose real path is looked for is rootfs, the directory is located outside of the system datastore and doesn't require a path to be looked for.

Details

• Affected Component: Virtualization Drivers
• Hypervisor: LXD
• Version: 5.8.0

Additional context
Add any other context about the problem here.

Progress Status

• Branch created
• Code committed to development branch
• Testing - QA
• Documentation
• Release notes - resolved issues, compatibility, known issues
• Code committed to upstream release/hotfix branches
• Documentation committed to upstream release/hotfix branches