unable to instantiate LXD container ver 5.8.1

[chronicom]

Description
I have followed the official documentation step by step to install the front end and compute node for LXD. I am not able to instantiate the LXD container.

To Reproduce
Front end:
ubuntu 18.04
opennebula 5.8.1
host file system: btrfs

Compute node:
ubuntu 18.04
opennebula 5.8.1
LXD 3.0 apt pakage
host file system: btrfs

storage backend:
NFS

datastores :
file datastore type: shared mode
image datastore type: qcow2
system datastore type: qcow2
each datastore under '/var/lib/one//datastores/' is symbolically linked to a directories created under '/mnt/NFS/'. The ownership of each directory is set to oneadmin
In the sunstone, the capacities of the datastores are correctly displayed.

LXD image:
'ubuntu_bionic - LXD' downloaded from 'linux containers'

Tried to instantiate the container from 'ubuntu_bionic - LXD' image with all default values without an NIC.

I am not sure if it is a bug or incorrect configuration of the directory permissions.

Expected behavior
A successful instantiation of LXD container

Details

• Affected Component: LXD node & storage
• Hypervisor: LXD
• Version: 5.8.1

Additional context
Add any other context about the problem here.
deployment log:

Sun Apr 28 21:54:04 2019 [Z0][VM][I]: New state is ACTIVE
Sun Apr 28 21:54:04 2019 [Z0][VM][I]: New LCM state is PROLOG
Sun Apr 28 21:54:07 2019 [Z0][VM][I]: New LCM state is BOOT
Sun Apr 28 21:54:07 2019 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/19/deployment.0
Sun Apr 28 21:54:09 2019 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_context.
Sun Apr 28 21:54:09 2019 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Sun Apr 28 21:54:13 2019 [Z0][VMM][I]: Command execution fail: cat << EOT | /var/tmp/one/vmm/lxd/deploy '/var/lib/one//datastores/108/19/deployment.0' 'compute2' 19 compute2
Sun Apr 28 21:54:13 2019 [Z0][VMM][I]: deploy: Using qcow2 mapper for /var/lib/one/datastores/108/19/disk.0
Sun Apr 28 21:54:13 2019 [Z0][VMM][I]: deploy: Mapping disk at /var/lib/lxd/storage-pools/default/containers/one-19/rootfs using device /dev/nbd0
Sun Apr 28 21:54:13 2019 [Z0][VMM][I]: deploy: Mounting /dev/nbd0 at /var/lib/lxd/storage-pools/default/containers/one-19/rootfs
Sun Apr 28 21:54:13 2019 [Z0][VMM][I]: deploy: Mapping disk at /mnt/NFS/108/19/mapper/disk.1 using device /dev/loop3
Sun Apr 28 21:54:13 2019 [Z0][VMM][I]: deploy: Mounting /dev/loop3 at /mnt/NFS/108/19/mapper/disk.1
Sun Apr 28 21:54:13 2019 [Z0][VMM][I]: deploy: Using qcow2 mapper for /var/lib/one/datastores/108/19/disk.0
Sun Apr 28 21:54:13 2019 [Z0][VMM][I]: deploy: Unmapping disk at /var/lib/lxd/storage-pools/default/containers/one-19/rootfs
Sun Apr 28 21:54:13 2019 [Z0][VMM][I]: /var/tmp/one/vmm/lxd/mapper.rb:203:in realpath': Permission denied @ realpath_rec - /var/lib/lxd/storage-pools/default/containers/one-19/rootfs (Errno::EACCES) Sun Apr 28 21:54:13 2019 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/mapper.rb:203:in unmap'
Sun Apr 28 21:54:13 2019 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/container.rb:359:in public_send' Sun Apr 28 21:54:13 2019 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/container.rb:359:in setup_disk'
Sun Apr 28 21:54:13 2019 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/container.rb:241:in block in setup_storage' Sun Apr 28 21:54:13 2019 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/container.rb:234:in each'
Sun Apr 28 21:54:13 2019 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/container.rb:234:in setup_storage' Sun Apr 28 21:54:13 2019 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/deploy:78:in rescue in

'
Sun Apr 28 21:54:13 2019 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/deploy:74:in `'
Sun Apr 28 21:54:13 2019 [Z0][VMM][I]: ExitCode: 1
Sun Apr 28 21:54:13 2019 [Z0][VMM][I]: Failed to execute virtualization driver operation: deploy.
Sun Apr 28 21:54:13 2019 [Z0][VMM][E]: Error deploying virtual machine
Sun Apr 28 21:54:13 2019 [Z0][VM][I]: New LCM state is BOOT_FAILURE

lxc.log:
lxc one-19 20190428195412.755 WARN conf - conf.c:lxc_setup_devpts:1616 - Invalid argument - F ailed to unmount old devpts instance
lxc one-19 20190428195412.784 ERROR start - start.c:start:2028 - No such file or directory - F ailed to exec "/sbin/init"
lxc one-19 20190428195412.784 ERROR sync - sync.c:__sync_wait:62 - An error occurred in anothe r process (expected sequence number 7)
lxc one-19 20190428195412.784 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:842 - Received container state "ABORTING" instead of "RUNNING"
lxc one-19 20190428195412.785 ERROR start - start.c:__lxc_start:1939 - Failed to spawn contain er "one-19"
lxc 20190428195412.786 WARN commands - commands.c:lxc_cmd_rsp_recv:132 - Connection reset by peer - Failed to receive response for command "get_state"

Progress Status

• Branch created
• Code committed to development branch
• Testing - QA
• Documentation
• Release notes - resolved issues, compatibility, known issues
• Code committed to upstream release/hotfix branches
• Documentation committed to upstream release/hotfix branches

[dann1]

Hello, we fixed a bug related to your specific setup, which is what happens here, however, it happens after the container fails to start. The log says LXD complains about missing the /sbin/init of the container image, which is another issue you are experiencing.

[dann1]

I'm closing this since the fix is already submitted, if you still experience the issue with the code of https://github.com/OpenNebula/one/tree/one-5.8 we could reopen it

[chronicom]

Hi thank you for the fix. I have found the problem with '/sbin/init'. It is because the image from 'linuxcontainers.org' is in raw format, and I downloaded into qcow2 datastore which messed up the internal of the file. I switched it back to shared mode and set the LXD_SECURITY_PRIVILEGED to "false". For some reason, I am still experiencing problem deploy the container. Do you think there might be some problem with my LXD configuration ?

Template:
CONTEXT = [
NETWORK = "YES",
SET_HOSTNAME = "$NAME",
SSH_PUBLIC_KEY = "$USER[SSH_PUBLIC_KEY]" ]
CPU = "1"
DISK = [
IMAGE_ID = "20" ]
GRAPHICS = [
LISTEN = "0.0.0.0",
TYPE = "vnc" ]
HYPERVISOR = "lxd"
LXD_SECURITY_PRIVILEGED = "false"
MEMORY = "1024"

Sunstone log:
Mon Apr 29 16:53:13 2019 [Z0][VM][I]: New state is ACTIVE
Mon Apr 29 16:53:13 2019 [Z0][VM][I]: New LCM state is PROLOG
Mon Apr 29 16:53:19 2019 [Z0][VM][I]: New LCM state is BOOT
Mon Apr 29 16:53:19 2019 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/23/deployment.0
Mon Apr 29 16:53:20 2019 [Z0][VMM][I]: Successfully execute transfer manager driver operation: tm_context.
Mon Apr 29 16:53:20 2019 [Z0][VMM][I]: Successfully execute network driver operation: pre.
Mon Apr 29 16:53:23 2019 [Z0][VMM][I]: Command execution fail: cat << EOT | /var/tmp/one/vmm/lxd/deploy '/var/lib/one//datastores/102/23/deployment.0' 'compute2' 23 compute2
Mon Apr 29 16:53:23 2019 [Z0][VMM][I]: deploy: Using raw filesystem mapper for /var/lib/one/datastores/102/23/disk.0
Mon Apr 29 16:53:23 2019 [Z0][VMM][I]: deploy: Mapping disk at /var/lib/lxd/storage-pools/default/containers/one-23/rootfs using device /dev/loop3
Mon Apr 29 16:53:23 2019 [Z0][VMM][I]: deploy: Resizing filesystem ext4 on /dev/loop3
Mon Apr 29 16:53:23 2019 [Z0][VMM][I]: deploy: Mounting /dev/loop3 at /var/lib/lxd/storage-pools/default/containers/one-23/rootfs
Mon Apr 29 16:53:23 2019 [Z0][VMM][I]: deploy: Mapping disk at /mnt/NFS/102/23/mapper/disk.1 using device /dev/loop4
Mon Apr 29 16:53:23 2019 [Z0][VMM][I]: deploy: Mounting /dev/loop4 at /mnt/NFS/102/23/mapper/disk.1
Mon Apr 29 16:53:23 2019 [Z0][VMM][I]: deploy: Using raw filesystem mapper for /var/lib/one/datastores/102/23/disk.0
Mon Apr 29 16:53:23 2019 [Z0][VMM][I]: deploy: Unmapping disk at /var/lib/lxd/storage-pools/default/containers/one-23/rootfs
Mon Apr 29 16:53:23 2019 [Z0][VMM][I]: deploy: Umounting disk mapped at /dev/loop3
Mon Apr 29 16:53:23 2019 [Z0][VMM][I]: deploy: Unmapping disk at /mnt/NFS/102/23/mapper/disk.1
Mon Apr 29 16:53:23 2019 [Z0][VMM][E]: deploy: Failed to detect block device from /mnt/NFS/102/23/mapper/disk.1
Mon Apr 29 16:53:23 2019 [Z0][VMM][E]: deploy: failed to dismantle container storage
Mon Apr 29 16:53:23 2019 [Z0][VMM][I]: /var/tmp/one/vmm/lxd/client.rb:145:in initialize': undefined method []' for #LXDError:0x0000563a7cdd6ed8 (NoMethodError)
Mon Apr 29 16:53:23 2019 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/deploy:86:in exception' Mon Apr 29 16:53:23 2019 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/deploy:86:in raise'
Mon Apr 29 16:53:23 2019 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/deploy:86:in rescue in <main>' Mon Apr 29 16:53:23 2019 [Z0][VMM][I]: from /var/tmp/one/vmm/lxd/deploy:74:in

'
Mon Apr 29 16:53:23 2019 [Z0][VMM][I]: ExitCode: 1
Mon Apr 29 16:53:23 2019 [Z0][VMM][I]: Failed to execute virtualization driver operation: deploy.
Mon Apr 29 16:53:23 2019 [Z0][VMM][E]: Error deploying virtual machine
Mon Apr 29 16:53:23 2019 [Z0][VM][I]: New LCM state is BOOT_FAILURE

lxc log
lxc one-23 20190429145323.891 ERROR conf - conf.c:mount_autodev:1131 - Permission denied - Failed to create "/dev" directory
lxc one-23 20190429145323.891 ERROR conf - conf.c:lxc_setup:3529 - Failed to mount "/dev"
lxc one-23 20190429145323.891 ERROR start - start.c:do_start:1263 - Failed to setup container "one-23"
lxc one-23 20190429145323.897 ERROR sync - sync.c:__sync_wait:62 - An error occurred in another process (expected sequence number 5)
lxc one-23 20190429145323.909 ERROR start - start.c:__lxc_start:1939 - Failed to spawn container "one-23"
lxc one-23 20190429145323.954 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:842 - Received container state "ABORTING" instead of "RUNNING"
lxc 20190429145323.105 WARN commands - commands.c:lxc_cmd_rsp_recv:132 - Connection reset by peer - Failed to receive response for command "get_state"

[dann1]

Well, the marketplace images are meant to be run with SECURITY_PRIVILEGED = true

[chronicom]

hmm could you please advice what should I do if I need to run unprivileged container ?

[dann1]

well, right now, the only choice would be to create your own image, since we are facing this issue

[chronicom]

Thank you very much. Looking forward to the next release :)