Incorrect /opt mount permission on 2605.8.0 on AWS

[ghost]

Description

We updated our AWS EC2 Kubernetes worker nodes to the latest AMI version, ami-006c88b6ca69f7811, which is version 2605.8.0. Our weave CNI plugin was not loading. When looking at the permissions of /opt we found it to be 711. Our weave plugin is expecting to reside in /opt/cni/bin

drwx--x--x. 3 root root 4096 Nov 19 21:18 opt

We reverted back to 2512.2.1 and opt is 755

drwxr-xr-x. 4 root root 4096 Nov 19 22:05 /opt

Impact

Kubernetes pods fail to launch due to weave not coming up

Environment and steps to reproduce

1. Set-up:

AWS: us-east-1
AMI: ami-006c88b6ca69f7811

2. Task:

Spin up EC2 instance on AMI ami-006c88b6ca69f7811

3. Action(s):

Spin up EC2 instance on AMI ami-006c88b6ca69f7811

4. Error:

/opt has incorrect permissions of 711

Expected behavior

/opt permissions should be 755

Additional information

N/A

[ghost]

Could this problem be why #278 exists?

[pothos]

Interesting, this folder is not created by Flatcar because it doesn't exist by default nor has a systemd-tmpfile directive to be created. In #176 you can see some ways how to find the process that does the modification. Can you try to find it out?

[zytrik]

I created a new EC2 instance with the referenced AMI with no userdata at all. @pothos to your point it looks like /opt is not part of the base image, it doesn't exist immediately on start up. However, without starting or placing any special software, after a minute or so the following directory tree appeared:

sudo ls -lRa /opt/
/opt/:
total 20
drwx--x--x.  3 root root 4096 Nov 20 15:39 .
drwxr-xr-x. 18 root root 4096 Nov 20 15:39 ..
drwx--x--x.  4 root root 4096 Nov 20 15:39 containerd

/opt/containerd:
total 32
drwx--x--x. 4 root root 4096 Nov 20 15:39 .
drwx--x--x. 3 root root 4096 Nov 20 15:39 ..
drwx--x--x. 2 root root 4096 Nov 20 15:39 bin
drwx--x--x. 2 root root 4096 Nov 20 15:39 lib

/opt/containerd/bin:
total 16
drwx--x--x. 2 root root 4096 Nov 20 15:39 .
drwx--x--x. 4 root root 4096 Nov 20 15:39 ..

/opt/containerd/lib:
total 16
drwx--x--x. 2 root root 4096 Nov 20 15:39 .
drwx--x--x. 4 root root 4096 Nov 20 15:39 ..

I suspect it's being triggered by the initial dockerd or containerd startup, but it's certainly related to the flatcar base configuration.

What's not too clear is what has changed in the last couple releases. Was there an existing /opt directory with appropriate permissions before? Are dockerd's default directory creation permissions different for some reason?

[zytrik]

Shutting down dockerd and containerd then starting them up individually, I was able to isolate the containerd startup as the action causing /opt to be created. This is approximately discussed here.

Workarounds aside, what's the best way to get back to having sane permissions on /opt by default? Does containerd need to be more thoughtful? Or do we need to have the directory back in the base image so we don't leave permissions up to whatever gets there first?

[pothos]

Thanks for tracking it down. I think containerd shouldn't create /opt with restrictive permissions but since it doesn't change existing permissions we can work around this by creating a systemd tmpfile directive.
Until it is shipped you can create your own and reboot:

$ cat /etc/tmpfiles.d/opt.conf
d   /opt                    0755    root    root    -   -

It will also adjust existing wrong permissions.

[pothos]

The fix will be part of the next releases in all channels, e.g., > 2605.8.0 for Stable.