[1.17+] Build does not push images to remote repository

[othercorey]

Expected behavior

Check for tag in remote repository and push images.

Actual behavior

Builds and tags images but does not push. Helm upgrade fails trying to pull image that doesn't exist.

Information

• Skaffold version: 1.17+
• Operating system: Windows 10
• Contents of skaffold.yaml:

apiVersion: skaffold/v2beta7
kind: Config

profiles:
- name: test
  build:
    local:
        push:
    tagPolicy:
      gitCommit:
        prefix: test-
    artifacts:
    - image: <account>.dkr.ecr.us-east-1.amazonaws.com/main-web-nginx
      context: .
      docker:
        dockerfile: deploy/docker/nginx.Dockerfile
    - image: <account>.dkr.ecr.us-east-1.amazonaws.com/main-web-phpfpm
      context: .
      docker:
        dockerfile: deploy/docker/php.Dockerfile
  deploy:
    kubeContext: prod-main
    helm:
      releases:
      - name: main-web
        chartPath: ./deploy/helm/main-web
        namespace: test
        artifactOverrides:
          nginx.image: <account>.dkr.ecr.us-east-1.amazonaws.com/main-web-nginx
          phpfpm.image: <account>.dkr.ecr.us-east-1.amazonaws.com/main-web-phpfpm

Steps to reproduce the behavior

1. skaffold run -p test

This has been working up to version 1.17. I'm not sure of any missing configuration requirements.

[MarlonGamez]

hi @othercorey, thanks for opening up this issue. Would it be possible for you to provide a small project setup to reproduce this? If you could also provide the output you're seeing here, that would be great!

[othercorey]

hi @othercorey, thanks for opening up this issue. Would it be possible for you to provide a small project setup to reproduce this? If you could also provide the output you're seeing here, that would be great!

I am working it, but it will take me a few days to setup. I won't be able to provide an AWS ECR account though. So, if it's somehow only happening with that repository, then will have to do further testing.

[MarlonGamez]

For now, if you could provide the output from skaffold run, that would be helpful too. We might be able to get some information about what's happening from that

[othercorey]

I omitted the docker image build logs.

Here is the build output with v1.19:

Generating tags...
 - <accountid>.dkr.ecr.us-east-1.amazonaws.com/main-web-phpfpm -> <accountid>.dkr.ecr.us-east-1.amazonaws.com/main-web-phpfpm:test-3.18.0-289-gd4737386
Checking cache...
 - <accountid>.dkr.ecr.us-east-1.amazonaws.com/main-web-phpfpm: Not found. Building
Found [prod-main] context, using local docker daemon.
Building [<accountid>.dkr.ecr.us-east-1.amazonaws.com/main-web-phpfpm]...
Successfully built 6d3465e2fe20
Successfully tagged <accountid>.dkr.ecr.us-east-1.amazonaws.com/main-web-phpfpm:test-3.18.0-289-gd4737386

Here is the build output with v1.16:

Generating tags...
 - <accountid>.dkr.ecr.us-east-1.amazonaws.com/main-web-phpfpm -> <accountid>.dkr.ecr.us-east-1.amazonaws.com/main-web-phpfpm:test-3.18.0-289-gd4737386
Checking cache...
 - <accountid>.dkr.ecr.us-east-1.amazonaws.com/main-web-phpfpm: Not found. Building
Building [<accountid>.dkr.ecr.us-east-1.amazonaws.com/main-web-phpfpm]...
Successfully built 6d3465e2fe20
Successfully tagged <accountid>.dkr.ecr.us-east-1.amazonaws.com/main-web-phpfpm:test-3.18.0-289-gd4737386
The push refers to repository [<accountid>.dkr.ecr.us-east-1.amazonaws.com/main-web-phpfpm]
5c9267607dc1: Pushed
9bc5ddad5e3e: Pushed
2c345fd97596: Pushed
2fdfdeb81619: Pushed
27c6d768710e: Layer already exists
0169fad74b88: Layer already exists
8af3c6bd6621: Layer already exists
1e3ca05d5151: Layer already exists
e7cd8cea3ee0: Layer already exists
862430d50960: Layer already exists
0574259d26c9: Layer already exists
9a7ad28c776d: Layer already exists
a08cf0e71e2a: Layer already exists
cfae40cc952a: Layer already exists
ec21dbbf635f: Layer already exists
f40b92b44d56: Layer already exists
836a2e2925cd: Layer already exists
7fe3caaea95c: Layer already exists
cd1253343b4e: Layer already exists
eab592dd1e81: Layer already exists
21e4fbf5d49d: Layer already exists
708ba1e65ea2: Layer already exists
cb42413394c4: Layer already exists
test-3.18.0-289-gd4737386: digest: sha256:a8267aef45cdfefbda09606cddac17bba1ce1b6ba1e870c4f75d78f7cfdc8d1b size: 5133
There is a new version (1.19.0) of Skaffold available. Download it from:
  https://github.com/GoogleContainerTools/skaffold/releases/tag/v1.19.0

[othercorey]

I created a test repository with a repro. However, you have to change the AWS ECR repository to something you have access to.

https://github.com/othercorey/skaffold-5337

This fails to push with --kube-context CLI option and local: push: in skaffold.yaml. My understanding is one of these should trigger the remote push.

skaffold 1.19

$ skaffold-119.exe build --kube-context prod-main -vdebug
time="2021-02-16T18:30:35-06:00" level=info msg="Skaffold &{Version:v1.19.0 ConfigVersion:skaffold/v2beta11 GitVersion: GitCommit:63949e28f40deed44c8f3c793b332191f2ef94e4 GitTreeState:dirty BuildDate:2021-01-28T17:30:48Z GoVersion:go1.14.2 Compiler:gc Platform:windows/amd64}"
time="2021-02-16T18:30:35-06:00" level=debug msg="Update check not enabled, skipping."
time="2021-02-16T18:30:35-06:00" level=info msg="Loaded Skaffold defaults from \"C:\\\\Users\\\\<name>\\\\.skaffold\\\\config\""
time="2021-02-16T18:30:35-06:00" level=debug msg="config version out of date: upgrading to latest \"skaffold/v2beta11\""
time="2021-02-16T18:30:35-06:00" level=debug msg="Defaulting deploy type to kubectl"
time="2021-02-16T18:30:35-06:00" level=info msg="Activated kube-context \"prod-main\""
time="2021-02-16T18:30:35-06:00" level=info msg="Using kubectl context: prod-main"
time="2021-02-16T18:30:35-06:00" level=debug msg="setting Docker user agent to skaffold-v1.19.0"
time="2021-02-16T18:30:35-06:00" level=debug msg="Using builder: local"
time="2021-02-16T18:30:35-06:00" level=debug msg="push value not present, defaulting to false because cluster.PushImages is false"
Generating tags...
 - <accountid>.dkr.ecr.us-east-1.amazonaws.com/skaffold-5337 -> time="2021-02-16T18:30:35-06:00" level=debug msg="Running command: [git describe --tags --always]"
time="2021-02-16T18:30:35-06:00" level=debug msg="Command output: [7dc9da1\n]"
time="2021-02-16T18:30:35-06:00" level=debug msg="Running command: [git status . --porcelain]"
time="2021-02-16T18:30:35-06:00" level=debug msg="Command output: []"
<accountid>.dkr.ecr.us-east-1.amazonaws.com/skaffold-5337:7dc9da1
time="2021-02-16T18:30:35-06:00" level=info msg="Tags generated in 39.9966ms"
Checking cache...
time="2021-02-16T18:30:35-06:00" level=debug msg="Found dependencies for dockerfile: [{test.txt / true}]"
time="2021-02-16T18:30:35-06:00" level=debug msg="push value not present, defaulting to false because cluster.PushImages is false"
 - <accountid>.dkr.ecr.us-east-1.amazonaws.com/skaffold-5337: Found. Tagging
time="2021-02-16T18:30:35-06:00" level=debug msg="push value not present, defaulting to false because cluster.PushImages is false"
time="2021-02-16T18:30:35-06:00" level=info msg="Cache check completed in 28.5141ms"
time="2021-02-16T18:30:35-06:00" level=debug msg="exporting metrics"
time="2021-02-16T18:30:36-06:00" level=debug msg="metrics uploading complete in 760.7898ms"

skaffold 1.16

$ skaffold build --kube-context prod-main -vdebug
time="2021-02-16T18:32:26-06:00" level=info msg="Skaffold &{Version:v1.16.0 ConfigVersion:skaffold/v2beta9 GitVersion: GitCommit:dcdf77cd94ec6a69d116c2123dc8727971e8b14c GitTreeState:clean BuildDate:2020-10-29T01:38:02Z GoVersion:go1.14.2 Compiler:gc Platform:windows/amd64}"
time="2021-02-16T18:32:26-06:00" level=info msg="Loaded Skaffold defaults from \"C:\\\\Users\\\\<name>\\\\.skaffold\\\\config\""
time="2021-02-16T18:32:26-06:00" level=info msg="Activated kube-context \"prod-main\""
time="2021-02-16T18:32:26-06:00" level=debug msg="Defaulting deploy type to kubectl"
time="2021-02-16T18:32:26-06:00" level=info msg="Using kubectl context: prod-main"
time="2021-02-16T18:32:26-06:00" level=debug msg="Using builder: local"
time="2021-02-16T18:32:26-06:00" level=debug msg="setting Docker user agent to skaffold-v1.16.0"
time="2021-02-16T18:32:26-06:00" level=debug msg="push value not present, defaulting to true because localCluster is false"
Generating tags...
 - <accountid>.dkr.ecr.us-east-1.amazonaws.com/skaffold-5337 -> time="2021-02-16T18:32:26-06:00" level=debug msg="Running command: [git describe --tags --always]"
time="2021-02-16T18:32:26-06:00" level=debug msg="Command output: [7dc9da1\n]"
time="2021-02-16T18:32:26-06:00" level=debug msg="Running command: [git status . --porcelain]"
time="2021-02-16T18:32:26-06:00" level=debug msg="Command output: []"
<accountid>.dkr.ecr.us-east-1.amazonaws.com/skaffold-5337:7dc9da1
time="2021-02-16T18:32:26-06:00" level=info msg="Tags generated in 40.0163ms"
Checking cache...
time="2021-02-16T18:32:26-06:00" level=debug msg="Found dependencies for dockerfile: [{test.txt / true}]"
 - <accountid>.dkr.ecr.us-east-1.amazonaws.com/skaffold-5337: Found. Pushing
time="2021-02-16T18:32:26-06:00" level=debug msg="Skipping credential configuration because docker-credential-gcloud is not on PATH."
The push refers to repository [<accountid>.dkr.ecr.us-east-1.amazonaws.com/skaffold-5337]
9a4c892ee8af: Layer already exists
85fcec7ef3ef: Layer already exists
3e5288f7a70f: Layer already exists
56bc37de0858: Layer already exists
1c91bf69a08b: Layer already exists
cb42413394c4: Layer already exists
7dc9da1: digest: sha256:1fefe39e32c7688014ab34e6c4668571486b756d63099c17074d7ff2584c5d88 size: 1569
time="2021-02-16T18:32:28-06:00" level=info msg="Cache check complete in 2.479962s"
There is a new version (1.20.0) of Skaffold available. Download it from:
  https://github.com/GoogleContainerTools/skaffold/releases/tag/v1.20.0

[othercorey]

Is there any more debug output I can provide? What triggers the check for a remote repo?

[aariacarterweir]

I'm seeing this too on GHCR. I've narrowed it down to using the --kube-context flag to a remote cluster, but having your current context in kubectl set to a local one.

[othercorey]

Comparing v1.16.0 to v1.17.0, the only really relevant change I can find is the options refactor:

#4986