Raspberry Pi issue - gateway didn't start

[schnable]

My actions before raising this issue

• [*] Followed the troubleshooting guide
• [*] Read/searched the docs
• [*] Searched past issues

I attempted to deploy openfaas using:

kubectl -f yaml_armhf

on my raspberry pi 4 cluster, but this results in the gateway pod entering a crashloop.

Expected Behaviour

gateway should start cleanly.

Current Behaviour

kubectl describe gateway-xxxxx-xxx

shows:

...
Warning  Failed     14m (x3 over 14m)     kubelet, k8s-worker-1  Error: failed to start container "faas-netes": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "chd
ir to cwd (\"/root\") set in config.json failed: permission denied": unknown
...

Examining the Dockerfile.armhf, I see this line: WORKDIR /root/

Examining gateway-dep.yaml, I see the following security context defined:

        securityContext:
          readOnlyRootFilesystem: true
          runAsUser: 10001

The latest version of raspian ships with the /root as mode 700. It looks to me like the user 10001 isn't able to change the work directory to /root which results in the startup failure. A quick look at the container showed:

 $ docker run --rm openfaas/gateway:0.17.0-armhf ls -l /root/
total 0
ls: can't open '/root/': Permission denied

As a work around, I commented out the "runAsUser: 10001" from the securityContext, and then the pod starts without any issues.

Possible Solution

I don't think running the pod as root is the best solution; I would recommend copying the user creation logic from the main Dockerfile to the armhf version; this would eliminate the need to have /root/ accessible to the pod, and would allow the pod to run without root escalation.

Steps to Reproduce (for bugs)

See above.

Context

Well - I was able to get things going with the work around. I might try to rebuild the docker container with the suggested fix and use this version rather than the official binary.

Your Environment

• FaaS-CLI version ( Full output from: faas-cli version ):
  not installed yet - just trying to get things isntalled for the first time.

• Docker version docker version (e.g. Docker 17.0.05 ):

19.03.2

• Are you using Docker Swarm or Kubernetes (FaaS-netes)?

K8s - 1.15.3

• Operating System and version (e.g. Linux, Windows, MacOS):

Linux raspberrypi3 4.19.66-v7+ openfaas/faas#1253 SMP Thu Aug 15 11:49:46 BST 2019 armv7l GNU/Linux

• Code example or link to GitHub repo or gist to reproduce problem:

See above

• Other diagnostic information / logs from troubleshooting guide

See above

Next steps

You may join Slack for community support.

[alexellis]

Hi @schnable,

Given that your version says 1.15, I'm assuming you're using kubeadm - what networking driver?

I have an RPi4 here with k3s (1.14) installed with k3sup.dev, it worked great and I would suggest you use the same. (I would advise you not to install Docker on the host if you are using k3s, it can cause an issue that looks similar to this.)

If your error applies only to the combination of the security context and K8s 1.15, then we may need to look the choice to specify a runAsUser.

Alex

pi@k4s-1:~ $ cat /etc/os-release 
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
pi@k4s-1:~ $ sudo uname -a
Linux k4s-1 4.19.58-v7l+ #1245 SMP Fri Jul 12 17:31:45 BST 2019 armv7l GNU/Linux
pi@k4s-1:~ $ sudo kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6-k3s.1", GitCommit:"4cd85f14854d942e9016cc15f399785c103242e9", GitTreeState:"clean", BuildDate:"2019-08-19T16:12+00:00Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/arm"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6-k3s.1", GitCommit:"4cd85f14854d942e9016cc15f399785c103242e9", GitTreeState:"clean", BuildDate:"2019-08-19T16:12+00:00Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/arm"}
pi@k4s-1:~ $ sudo kubectl get pod -n openfaas
NAME                            READY   STATUS    RESTARTS   AGE
alertmanager-6ddc5bdff8-btvcd   1/1     Running   0          17m
faas-idler-59dfd85f6c-l9lm8     1/1     Running   2          17m
gateway-d546b8644-rfmm7         2/2     Running   0          17m
nats-d4c9d8d95-pps45            1/1     Running   0          17m
prometheus-9c99c4499-xmpqv      1/1     Running   0          17m
queue-worker-5595b6b8c6-j6p6s   1/1     Running   0          17m
pi@k4s-1:~ $ 

[alexellis]

Examining the Dockerfile.armhf, I see this line: WORKDIR /root/

Where did you find that? I don't see the gateway using the /root directory?

WORKDIR /home/app

Dockerfile.armhf

Here's the live container:

sudo kubectl exec -n openfaas -ti gateway-d546b8644-rfmm7 /bin/sh
Defaulting container name to gateway.
Use 'kubectl describe pod/gateway-d546b8644-rfmm7 -n openfaas' to see all of the containers in this pod.
/home/app $ pwd
/home/app
/home/app $ ls
assets   gateway
/home/app $ 

[schnable]

Thanks for the response.

I am using k8s directly (rather than k3s) as I am trying to create a specific configuration. I am currently running with flannel.

WRT k3s-io/k3s#703, I suspect this is more of an issue with the new debian iptables implementation - see kubernetes/kubernetes#71305. There is workaround to force all the containers to use the legacy iptables which is what I am doing.

I see that you recently updated the repository and pulled in a new version of the dockerfile that has the changes I suggested - this is great. To answer your question "Where did you find that" - see: https://github.com/openfaas/faas-netes/blob/b89240394f46b30bd83ba9f75dd1f93cf680113a/Dockerfile.armhf - the version from just a few days ago.

The container I was using had a release tag of 0.8.5 and I see you released 0.8.6 just a little while ago with this fix. Thanks. I will give it a try.

[alexellis]

/close: fix published

[alexellis]

It was confusing when you said "gateway" is running as "root" and then mentioned WORKDIR /root/ because that's not the case for the gateway for either platform.

Turned out that faas-netes for x86_64 was also not root, but the override Dockerfile.armhf still had it from some time ago.

By "k8s directly" I assume you mean kubeadm right?

Please let me know how you get on with 0.8.6.

[schnable]

Sorry for the confusion. I was thinking that when the container's Dockerfile doesn't specify USER, and when you don't have a runAsUser specified in your deployment's security context (which my hacky workaround was doing), you end up running as root.

Yes - I am creating the cluster using kubeadm. I've just retested using the 0.8.6 and all is working well now. Thanks.

[alexellis]

That's great news and thank you for logging the issue and being so helpful with the level of detail.

Are there any updates needed to my guide over here for kubeadm and 1.15? https://github.com/teamserverless/k8s-on-raspbian

Please feel free to say hi on Slack to talk about more armhf + OpenFaaS

[alexellis]

/msg: slack

[derek]

--
Join Slack to connect with the community
https://docs.openfaas.com/community