Fail to run task with multi-arch image on non amd64 cluster

[barthy1]

Expected Behavior

Task with multi-arch image should be successfully executable on non amd64 architetures.

Actual Behavior

Task failed for s390x and ppc64le architectures with error

Error executing command: fork/exec /bin/bash: exec format error

Research of the pod logs showed that instead of pulling correct version of multi-arch image for appropriate architecture(s390x or ppc64le), always amd64 version is pulled. amd64 container image cannot run on another archirectire, as a result - exec format error

# kubectl describe pod wd-task-run-pod-9kbqj
Name:         wd-task-run-pod-9kbqj
Namespace:    default
Priority:     0
Node:         node3/192.168.122.29
Start Time:   Tue, 11 Aug 2020 11:54:04 +0200
Labels:       app.kubernetes.io/managed-by=tekton-pipelines
              tekton.dev/task=wd-task
              tekton.dev/taskRun=wd-task-run
Annotations:  pipeline.tekton.dev/release: devel
              tekton.dev/ready: READY
Status:       Failed
IP:           10.144.2.230
IPs:
  IP:           10.144.2.230
Controlled By:  TaskRun/wd-task-run
Init Containers:
  place-tools:
    Container ID:  docker://1799df7db1b9c82b41a4d38991b6cf67e032361aeee495a603a91ca411b6aea6
    Image:         sys-boe-dev-team-docker-local.artifactory.swg-devops.com/github.com/tektoncd/pipeline/cmd/entrypoint:test3@sha256:54b1c40d5769b0b4c69ae35eeb5b2a188f203acf8251980d999b1d548c502574
    Image ID:      docker-pullable://sys-boe-dev-team-docker-local.artifactory.swg-devops.com/github.com/tektoncd/pipeline/cmd/entrypoint@sha256:54b1c40d5769b0b4c69ae35eeb5b2a188f203acf8251980d999b1d548c502574
    Port:          <none>
    Host Port:     <none>
    Command:
      cp
      /ko-app/entrypoint
      /tekton/tools/entrypoint
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 11 Aug 2020 11:54:05 +0200
      Finished:     Tue, 11 Aug 2020 11:54:05 +0200
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /tekton/tools from tekton-internal-tools (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-dx8kz (ro)
Containers:
  step-test:
    Container ID:  docker://0441ab2704b2c479da5392fd3fe16460b7dbd7238c6eccc369e5603dad0196fa
    Image:         index.docker.io/library/ubuntu@sha256:60f560e52264ed1cb7829a0d59b1ee7740d7580e0eb293aca2d722136edb1e24
    Image ID:      docker-pullable://ubuntu@sha256:60f560e52264ed1cb7829a0d59b1ee7740d7580e0eb293aca2d722136edb1e24
    Port:          <none>
    Host Port:     <none>
    Command:
      /tekton/tools/entrypoint
    Args:
      -wait_file
      /tekton/downward/ready
      -wait_file_content
      -post_file
      /tekton/tools/0
      -termination_path
      /tekton/termination
      -entrypoint
      /bin/bash
      --
      -c
      echo YES
    State:          Terminated
      Reason:       Error
      Message:      [{"key":"StartedAt","value":"2020-08-11T09:54:08.477Z","resourceRef":{}}]
      Exit Code:    1
      Started:      Tue, 11 Aug 2020 11:54:06 +0200
      Finished:     Tue, 11 Aug 2020 11:54:08 +0200
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:                0
      ephemeral-storage:  0
      memory:             0
    Environment:
      HOME:  /tekton/home
    Mounts:
      /tekton/creds from tekton-creds-init-home-pzxtp (rw)
      /tekton/downward from tekton-internal-downward (rw)
      /tekton/home from tekton-internal-home (rw)
      /tekton/results from tekton-internal-results (rw)
      /tekton/tools from tekton-internal-tools (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-dx8kz (ro)
      /workspace from tekton-internal-workspace (rw)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  tekton-internal-workspace:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  tekton-internal-home:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  tekton-internal-results:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  tekton-internal-tools:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  tekton-internal-downward:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.annotations['tekton.dev/ready'] -> ready
  tekton-creds-init-home-pzxtp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  default-token-dx8kz:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-dx8kz
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  28m   default-scheduler  Successfully assigned default/wd-task-run-pod-9kbqj to node3
  Normal  Pulled     28m   kubelet, node3     Container image "sys-boe-dev-team-docker-local.artifactory.swg-devops.com/github.com/tektoncd/pipeline/cmd/entrypoint:test3@sha256:54b1c40d5769b0b4c69ae35eeb5b2a188f203acf8251980d999b1d548c502574" already present on machine
  Normal  Created    28m   kubelet, node3     Created container place-tools
  Normal  Started    28m   kubelet, node3     Started container place-tools
  Normal  Pulled     28m   kubelet, node3     Container image "index.docker.io/library/ubuntu@sha256:60f560e52264ed1cb7829a0d59b1ee7740d7580e0eb293aca2d722136edb1e24" already present on machine
  Normal  Created    28m   kubelet, node3     Created container step-test
  Normal  Started    28m   kubelet, node3     Started container step-test

Steps to Reproduce the Problem

1. Use s390x k8s cluster and install Tekton pipeline on top.
2. Create Task

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: wd-task
spec:
  steps:
  - args:
    - -c
    - echo YES
    image: ubuntu
    name: "test"
    resources: {}
    workingDir: /HELLOMOTO

3. Create TaskRun

apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  labels:
    app.kubernetes.io/managed-by: tekton-pipelines
    tekton.dev/task: wd-task
  name: wd-task-run
spec:
  serviceAccountName: default
  taskRef:
    kind: Task
    name: wd-task
  timeout: 1h0m0s

4. Get TaskRun status -

NAME          SUCCEEDED   REASON   STARTTIME   COMPLETIONTIME
wd-task-run   False       Failed   7m42s       7m35s

Additional Info

This issue is open as part of initiative to enable Tekton for non amd64 architectures, specifically s390x - #3064

• Kubernetes version:

  Output of kubectl version:

# kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-16T14:17:52Z", GoVersion:"go1.13.13", Compiler:"gc", Platform:"linux/s390x"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-26T03:39:24Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/s390x"}

• Tekton Pipeline version:

Pipeline was installed to s390x k8s cluster using ko apply -f config/ from master branch as per https://github.com/tektoncd/pipeline/blob/3a7a693429c605a67e30efa5b74675e403be79bb/DEVELOPMENT.md#install-pipeline

[barthy1]

cc @davidhay1969