Initializing containers fails on self-hosted runners

[yisiliu]

Describe the bug
Our self hosted actions runner stopped working correctly during the step "Initialize Containers" and threw out an error message Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.40/version: dial unix /var/run/docker.sock: connect: permission denied. It was first observed in https://github.com/DimensionDev/Maskbook/runs/669343239?check_suite_focus=true.

We migrated our almost all CI jobs to actions on our self hosted runner in this PR and it worked just fine.

To Reproduce
I don't know if this can be reproduced on other machines but I can provide the specs of our machine:

Ubuntu 18.04 on AWS
4 Core 8GB Mem

Expected behavior
It initializes the container correctly.

Runner Version and Platform

Neither 2.165.2 nor 2.262.1 works
Ubuntu 18.04

What's not working?

[TingluoHuang]

@yisiliu make sure your runner's run-as account has permission to execute docker command without sudo
https://docs.docker.com/engine/install/linux-postinstall/

[yisiliu]

@TingluoHuang Thanks for the reply! Actually it should work without sudo since otherwise no jobs could run without throwing this error. My current observation is that if multiple workflows run simultaneously, the jobs in both workflows would fail at initialization. Also, jobs cannot run in parallel either, so I added dependencies between steps to force them to run sequentially, e.g. https://github.com/dimensiondev/maskbook/actions/runs/109387215. Do you have any idea why this is happening? One possible reason is that all the jobs require the same docker image but actions runner would start each instance in parallel and expose them to different ports, wouldn't it? Would some kinds of lock cause such permission denied errors?

[yisiliu]

Can we possibly receive any help here? Or we will need to manually re-run the failing workflows due to the conflicts of parallel runs of workflows.

[fidanf]

We've been encountering this problem earlier this week.
Migrating to Ubuntu focal seems to have fix it.
Then we had to adjust docker privileges and restart the runner process (see this link)

[yisiliu]

We've been encountering this problem earlier this week.

Migrating to Ubuntu focal seems to have fix it.

Then we had to adjust docker privileges and restart the runner process (see this link)

Thanks! I'll take a look and let you know if it fixes our issue

[nikola-jokic]

Hi @yisiliu,

Are you still seeing this issue? I just want to confirm if this is still an issue for you?

[yisiliu]

@nikola-jokic hi thanks for the feedback! Actually since this issue was never handled, we decided to switch back to circle ci and haven't given it another try to test it out. I'd probably test it later this week and let you know.

[nikola-jokic]

Great! Thanks, @yisiliu!

[nikola-jokic]

Hey @yisiliu,

I will close this issue now until we hear back from you. If you confirm that this issue is still occurring, feel free to reopen it and we will take a look!