Pod startup failing due to new startup.sh

[rich-bain]

Controller Version

0.24.1

Helm Chart Version

0.19.1

CertManager Version

1.8.1

Deployment Method

Other

cert-manager installation

• Installation process with Helm Via flux. Tested and working correctly with custom domain.

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue

Resource Definitions

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: default-org-runners
  namespace: actions-runner-system
spec:
  replicas: 1
  template:
    spec:
      image: summerwind/actions-runner-dind # <-- was unpinned
      dockerdWithinRunnerContainer: true
      organization: <redacted>
      env: []

To Reproduce

Try to run on a GKE cluster with the latest image `latest@sha256:331d12b3c2bb35436f82dccfc661afd13bb093afd3461d19cc5fce74b64e896c`. Pod correctly schedules but `runner`

Describe the bug

See textPayload key below.

{
  "textPayload": "tee: 'standard output': Bad file descriptor",    
  "insertId": "vrxebg7zrlowahpw",
  "resource": {
    "type": "k8s_container",
    ...
    }
  },
  "timestamp": "2022-06-23T03:45:09.572895611Z",
  "severity": "ERROR",
  "labels": {
       ...
  },
   ...
}

This is a k8s_container resource which means the pod was correctly scheduled but failed during startup.

Symptoms are a restart loop of pods every ~1 second.

Describe the expected behavior

Pod should start. Changing nothing but going back to summerwind/actions-runner-dind:v2.293.0-ubuntu-20.04-933b0c7@sha256:635aa33ed5fc83f5df7a27986f654500fc28eeb619498888f3442a133b54258b fixes the issue.

Controller Logs

https://gist.github.com/rich-bain/440988ac221992e196fba6aa4faeb711

Runner Pod Logs

Unavailable due to immediate deletion.

See stackdriver log (only a single item produced):
https://gist.github.com/rich-bain/440988ac221992e196fba6aa4faeb711

Additional Context

Worked fine yesterday. Spun up some new nodes which invalided my docker cache. Pinning back to the old version fixes the issue.

Issue either https://github.com/actions-runner-controller/actions-runner-controller/blob/master/runner/startup.sh#L30 or https://github.com/actions-runner-controller/actions-runner-controller/blob/master/runner/startup.sh#L45

[shaikatz]

Seeing the same behavior, the pod logs shows: tee: 'standard output': Bad file descriptor

[ChrisBr]

@rich-bain can you show which version you pinned to?

[ChrisBr]

Pinning the image to

image: summerwind/actions-runner-dind:v2.293.0-ubuntu-20.04-933b0c7@sha256:635aa33ed5fc83f5df7a27986f654500fc28eeb619498888f3442a133b54258b

resolved the issue for us!

[kyontan]

We faced the same error on new image actions-runner-dind:v2.293.0-ubuntu-20.04.
helm chart version: 0.19.1

The error log is one line:

environment: AWS EKS v1.22.9-eks-a64ea69 with Karpenter (containerd).

tee: 'standard output': Bad file descriptor

We've already reverted actions-runner-dind image version to v2.293.0-ubuntu-20.04 and everything goes back well.

[mattpopa]

Getting the same issue, reverted to 2.293.0-ubuntu-20.04-933b0c7