Image pull fail on helm deploy when sync is enabled

[yoshwata]

Expected behavior

Image pull as the settings of values.yaml on skaffold deploy.
skaffold dev is no problem.

Actual behavior

ImagePullBackOff occurs to pull image name like skaffold-helm:skaffold-helm

When I try to sync with this code(This is also put as reproduction example too), syncing works well.

Logs of skaffold dev

$ ../../out/skaffold-darwin-amd64 dev --default-repo docker.io/yoshwata Generating tags... - docker.io/yoshwata/skaffold-helm -> docker.io/yoshwata/skaffold-helm:latest Tags generated in 97.378µs Starting build... Building [docker.io/yoshwata/skaffold-helm]... Sending build context to Docker daemon 4.096kB Step 1/2 : FROM nginx:stable ---> 18ff9f3390d6 Step 2/2 : COPY static /usr/share/nginx/html/ ---> 6018f43d17f9 Successfully built 6018f43d17f9 Successfully tagged yoshwata/skaffold-helm:latest The push refers to repository [docker.io/yoshwata/skaffold-helm] f16acba8faf9: Preparing 20435c24eb60: Preparing 7c6961c07bf2: Preparing 6270adb5794c: Preparing 6270adb5794c: Layer already exists 7c6961c07bf2: Layer already exists 20435c24eb60: Layer already exists f16acba8faf9: Pushed latest: digest: sha256:cbb7c452b0d812e82300b833c201049e588202b9e819469c7e414eb2bf156712 size: 1155 Build complete in 16.122171527s Starting test... Test complete in 6.802µs Starting deploy... Error: release: "skaffold-helm" not found Helm release skaffold-helm not installed. Installing... No requirements found in skaffold-helm/charts. NAME: skaffold-helm LAST DEPLOYED: Fri Aug 23 11:14:12 2019 NAMESPACE: default STATUS: DEPLOYED

RESOURCES:
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
skaffold-helm-56898c9f46-g6gbp 0/1 Pending 0 0s

==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
skaffold-helm-skaffold-helm ClusterIP 10.43.15.1 80/TCP 0s

==> v1beta1/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
skaffold-helm 0/1 1 0 0s

==> v1beta1/Ingress
NAME HOSTS ADDRESS PORTS AGE
skaffold-helm-skaffold-helm * 80 0s

Deploy complete in 922.265861ms
Watching for changes...
Syncing 1 files for docker.io/yoshwata/skaffold-helm:latest@sha256:cbb7c452b0d812e82300b833c201049e588202b9e819469c7e414eb2bf156712
Watching for changes...

However skaffold deploy does not work at this condition because of ImagePullBackOff.

Logs of skaffold deploy

$ ../../out/skaffold-darwin-amd64 deploy --default-repo docker.io/yoshwata Starting deploy... Error: release: "skaffold-helm" not found Helm release skaffold-helm not installed. Installing... No requirements found in skaffold-helm/charts. NAME: skaffold-helm LAST DEPLOYED: Fri Aug 23 11:15:39 2019 NAMESPACE: default STATUS: DEPLOYED

RESOURCES:
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
skaffold-helm-56898c9f46-g6gbp 0/1 Terminating 0 87s
skaffold-helm-5dfc67475-k85vg 0/1 Pending 0 0s

==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
skaffold-helm-skaffold-helm ClusterIP 10.43.100.130 80/TCP 0s

==> v1beta1/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
skaffold-helm 0/1 1 0 0s

==> v1beta1/Ingress
NAME HOSTS ADDRESS PORTS AGE
skaffold-helm-skaffold-helm * 80 0s

Deploy complete in 1.084552247s
There is a new version (0.36.0) of Skaffold available. Download it at https://storage.googleapis.com/skaffold/releases/latest/skaffold-darwin-amd64

Result of kubectl get po

NAME                            READY   STATUS             RESTARTS   AGE
skaffold-helm-5dfc67475-k85vg   0/1     ImagePullBackOff   0          24s
svclb-node-4j2j8                1/1     Running            0          47d
svclb-node-74b9w                1/1     Running            0          47d

Image name is wrong, that should be nginx:stable as it is specified at values.yaml.

$ k get po skaffold-helm-5dfc67475-k85vg -o=jsonpath='{.spec.containers[0].image}'
skaffold-helm:skaffold-helm

Information

• Skaffold version: version v0.33.0-357-g706469d9-dirty
• Operating system: mac osx
• Contents of skaffold.yaml:

apiVersion: skaffold/v1beta13
kind: Config
build:
  tagPolicy:
    sha256: {}
  artifacts:
  - image: skaffold-helm
    sync:
      manual:
      - src: 'static/**/*.*'
        dest: '/usr/share/nginx/html'
        strip: static/
deploy:
  helm:
    releases:
    - name: skaffold-helm
      chartPath: skaffold-helm
      #wait: true
      #valuesFiles:
      #- helm-skaffold-values.yaml
      values:
        # image: gcr.io/k8s-skaffold/skaffold-helm
        image: skaffold-helm
      #recreatePods will pass --recreate-pods to helm upgrade
      #recreatePods: true
      #overrides builds an override values.yaml file to run with the helm deploy
      #overrides:
      # some:
      #   key: someValue
      #setValues get appended to the helm deploy with --set.
      #setValues:
        #some.key: someValue
      imageStrategy:
        helm: {}

Steps to reproduce the behavior

1. checkout this sample
2. skaffold deploy --default-repo docker.io/yoshwata
3. deployment can not pull image

What I guess

This line might be something wrong.

[dgageot]

Hi @yoshwata, I wonder if there's any way you can test #2772 and see if solves your issue.

[yoshwata]

Thank you @dgageot. It seems that the problem still remaining.

yoshwata ex 0 at MBP15UAW-B001 in ~/git/githubcom/GoogleContainerTools/skaffold/examples/helm-deployment-hot-reload on helm-image-pull-error
$ ../../out/skaffold-darwin-amd64 deploy --default-repo docker.io/yoshwata
Starting deploy...
No requirements found in skaffold-helm/charts.
Release "skaffold-helm" has been upgraded.
LAST DEPLOYED: Sat Aug 31 18:00:56 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Pod(related)
NAME                           READY  STATUS   RESTARTS  AGE
skaffold-helm-5dfc67475-nlw5j  0/1    Pending  0         0s

==> v1/Service
NAME                         TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)  AGE
skaffold-helm-skaffold-helm  ClusterIP  10.43.100.130  <none>       80/TCP   8d

==> v1beta1/Deployment
NAME           READY  UP-TO-DATE  AVAILABLE  AGE
skaffold-helm  0/1    1           0          0s

==> v1beta1/Ingress
NAME                         HOSTS  ADDRESS  PORTS  AGE
skaffold-helm-skaffold-helm  *      80       8d


Deploy complete in 1.21004526s

yoshwata ex 0 at MBP15UAW-B001 in ~/git/githubcom/GoogleContainerTools/skaffold/examples/helm-deployment-hot-reload on helm-image-pull-error
$ k get po
NAME                            READY   STATUS         RESTARTS   AGE
skaffold-helm-5dfc67475-nlw5j   0/1     ErrImagePull   0          10s
svclb-node-4j2j8                1/1     Running        0          55d
svclb-node-74b9w                1/1     Running        0          55d

yoshwata ex 0 at MBP15UAW-B001 in ~/git/githubcom/GoogleContainerTools/skaffold/examples/helm-deployment-hot-reload on helm-image-pull-error

yoshwata ex 0 at MBP15UAW-B001 in ~/git/githubcom/GoogleContainerTools/skaffold/examples/helm-deployment-hot-reload on helm-image-pull-error
$ ../../out/skaffold-darwin-amd64 version
v0.32.0-725-g1f36cca5

[dgageot]

Hi @yoshwata I'm afraid skaffold deploy only works with --images for each artifact.
Since skaffold dev is working for you, I think everything is configured well.
Maybe what you are looking for, for one-shot deployments, is skaffold run.

[yoshwata]

Hi @dgageot. I tried skaffold run and it works well. This means that it works well if pipeline has pre-builded artifact. My ideal is that pipeline uses value of values.yaml or setValues: {} of skaffold.yaml as image name when there is no pre-build image, but now it is thrown away because I specify it like this.

[dgageot]

Hi @yoshwata, I'm not sure I understand what you're saying. I'll try to clarify what I meant.

skaffold run and skaffold deploy are two ways to deploy your application.

• skaffold run will build the artifacts, tag them, update your k8s manifests with those tags and send the manifests to a Kubernetes cluster.
• skaffold deploy does a bit less. It assumes you have built the artefacts before, any way you want, and that you pass the tags with one --images TAG per artifact.

Both use cases work with the exact same skaffold.yaml and manifests.
skaffold run is the preferred, simpler way of doing.
skaffold deploy is very specific, more complicated to use.

In both cases, since you are using helm, you have to use the values section of the skaffold.yaml to map the image names to arbitrary keys that can be used in helm templates. Something like this

[yoshwata]

@dgageot Thank you for explanation. I understood and there is no problem for me. Let me close this issue.