Self Hosted Runner running in a container fails to run steps inside a service container "sh: 0: Can't open /__w/_temp/xxx.sh"

[dimisjim]

Describe the bug
Hey,

My self-hosted runner is basically a container in itself that has access to the docker daemon socket. This allow it to perform docker cli commands, useful for all kinds of operations. For simple stuff it's quite neat and works as expected.

Now, I am trying to create a workflow that starts up a container and then tries to run its steps inside that container.

But it seems that it can't even load some simple bash "run" steps such as echo "test" and returns this instead:

Run echo "test"
  echo "test"
  shell: sh -e {0}
sh: 0: Can't open /__w/_temp/8e725f1d-7b7b-42e6-b0ad-103fa9200580.sh
Error: Process completed with exit code 127.

If I define the shell parameter with value bash in the run step, I get something slightly different:

bash: /__w/_temp/baf2900f-87a1-4447-8b71-965fe79bef7d.sh: No such file or directory
Error: Process completed with exit code 1.

To Reproduce
Here's my yaml section in question:

.....
build-environment:
    needs: get-ecr-credentials
    runs-on: [self-hosted, gh-medium-x86]

    container: 
      image: <my-custom-image-in-ecr>
      credentials:
        username: AWS
        password: ${{needs.get-ecr-credentials.outputs.password}}
      options: >-
        --log-driver journald
        --log-opt tag=mytag

    steps:
    - name: Test Run
      run: echo "yeah"

Expected behavior
Runner runs the echo step inside my custom image without erroring out

Runner Version and Platform

myoung34/github-runner:2.276.1-ubuntu-bionic

Amazon Linux 2 x86_64

---

EDIT 16/2:

Tried with setting privileged=true for the runner container, got the same errors

Tried also to use another non-custom container straight from the docker hub (i.e ubuntu:20.04), got the same errors

[djcarpe]

Did you ever find a solution to this?

[dimisjim]

@djcarpe This is caused due to the lack of support for running the runner application inside docker: #406

[djcarpe]

Thank you! Exactly what I needed to see.

[gibsonje]

I can't find any workaround for this

[AllanOricil]

How does ARC get over it? I don't get it
https://github.com/actions/actions-runner-controller

[prein]

Not sure if it helps but mounting the workdir from the host using dockerVolumeMounts in the ARC RunnerDeployment helps

[SirCosty]

hross You mark it as duplicate.
So where is the duplicate? Please be more concise.

[SirCosty]

@djcarpe could you find a solution to this? Which was the issue?

[KrzysztofUstowski]

I solved this problem for myself, I'm leaving a comment for others looking for a solution:
You need to make sure that "docker-dind" and "docker-runner" share the "/home/runner" directory, only then the directory mappings between the runner and the container running inside the runner will match

[kdmkone]

I solved this problem for myself, I'm leaving a comment for others looking for a solution: You need to make sure that "docker-dind" and "docker-runner" share the "/home/runner" directory, only then the directory mappings between the runner and the container running inside the runner will match

@KrzysztofUstowski Do the folders have to be the ones of the running container or just the raw definitions?

[KrzysztofUstowski]

@kdmkone

Your runner, as part of executing an action will create a temporary container (to execute the commands defined in the job) and it will try to map its own working directory to this newly created temporary container.

The problem apears when your runner itself is embedded in a container (using dind), because when your runner tries to set the mapping for the newly created temporary container (this mapping between himself and the temp container), it actually sends the mapping request to its docker host (dind). So, if the dind container does not have access to runners /home/runner directory then dind cannot really set the mappings correctly and finally is pointing to the empty directory.

Thats why you see Can't open /__w/_temp/8e725f1d-7b7b-42e6-b0ad-103fa9200580.sh because /__w/_temp/ (inside temp container) is pointing to /home/runner/_work/_temp/ and this is not the path inside the runner container as runner expect, but actually inside the dind container (and dind doesn't have a /home/runner directory, so its link to empty dir).

Solution: dind container and runner container have to see, read and write the same directory /home/runner, you can achieve that by sharing the same named volume between containers, or share the same host volume.

My example of docker-compose.yml

services:

  docker_worker_dev_1:
    image: docker:27.3-dind
    restart: always
    privileged: true
    environment:
      - TZ=Europe/Warsaw
    volumes:
      - ./worker_dev_1/_certs:/certs
      - ./worker_dev_1/_var_lib_docker:/var/lib/docker
      - ./worker_dev_1/home:/home/runner
      
  worker_dev_1:
    image: ghcr.io/actions/actions-runner:2.319.1
    restart: always
    environment:
      - TZ=Europe/Warsaw
      - DOCKER_HOST=tcp://docker_worker_dev_1:2376/
      - DOCKER_TLS=1
    depends_on:
      - docker_worker_dev_1
    volumes:
      - ./worker_dev_1/_certs/client/ca.pem:/home/runner/.docker/ca.pem
      - ./worker_dev_1/_certs/client/cert.pem:/home/runner/.docker/cert.pem
      - ./worker_dev_1/_certs/client/key.pem:/home/runner/.docker/key.pem
      - ./worker_dev_1/_var_lib_docker:/var/lib/docker
      - ./worker_dev_1/home:/home/runner
    command: sh -c "./run.sh"