[1.17+] Build does not push images to remote repository #5334

othercorey · 2021-02-02T14:47:13Z

Expected behavior

Check for tag in remote repository and push images.

Actual behavior

Builds and tags images but does not push. Helm upgrade fails trying to pull image that doesn't exist.

Information

Skaffold version: 1.17+
Operating system: Windows 10
Contents of skaffold.yaml:

apiVersion: skaffold/v2beta7
kind: Config

profiles:
- name: test
  build:
    local:
        push:
    tagPolicy:
      gitCommit:
        prefix: test-
    artifacts:
    - image: <account>.dkr.ecr.us-east-1.amazonaws.com/main-web-nginx
      context: .
      docker:
        dockerfile: deploy/docker/nginx.Dockerfile
    - image: <account>.dkr.ecr.us-east-1.amazonaws.com/main-web-phpfpm
      context: .
      docker:
        dockerfile: deploy/docker/php.Dockerfile
  deploy:
    kubeContext: prod-main
    helm:
      releases:
      - name: main-web
        chartPath: ./deploy/helm/main-web
        namespace: test
        artifactOverrides:
          nginx.image: <account>.dkr.ecr.us-east-1.amazonaws.com/main-web-nginx
          phpfpm.image: <account>.dkr.ecr.us-east-1.amazonaws.com/main-web-phpfpm

Steps to reproduce the behavior

skaffold run -p test

This has been working up to version 1.17. I'm not sure of any missing configuration requirements.

MarlonGamez · 2021-02-02T18:28:13Z

hi @othercorey, thanks for opening up this issue. Would it be possible for you to provide a small project setup to reproduce this? If you could also provide the output you're seeing here, that would be great!

othercorey · 2021-02-04T03:47:52Z

hi @othercorey, thanks for opening up this issue. Would it be possible for you to provide a small project setup to reproduce this? If you could also provide the output you're seeing here, that would be great!

I am working it, but it will take me a few days to setup. I won't be able to provide an AWS ECR account though. So, if it's somehow only happening with that repository, then will have to do further testing.

MarlonGamez · 2021-02-08T20:41:12Z

For now, if you could provide the output from skaffold run, that would be helpful too. We might be able to get some information about what's happening from that

othercorey · 2021-02-10T07:11:28Z

I omitted the docker image build logs.

Here is the build output with v1.19:

Generating tags...
 - <accountid>.dkr.ecr.us-east-1.amazonaws.com/main-web-phpfpm -> <accountid>.dkr.ecr.us-east-1.amazonaws.com/main-web-phpfpm:test-3.18.0-289-gd4737386
Checking cache...
 - <accountid>.dkr.ecr.us-east-1.amazonaws.com/main-web-phpfpm: Not found. Building
Found [prod-main] context, using local docker daemon.
Building [<accountid>.dkr.ecr.us-east-1.amazonaws.com/main-web-phpfpm]...
Successfully built 6d3465e2fe20
Successfully tagged <accountid>.dkr.ecr.us-east-1.amazonaws.com/main-web-phpfpm:test-3.18.0-289-gd4737386

Here is the build output with v1.16:

Generating tags...
 - <accountid>.dkr.ecr.us-east-1.amazonaws.com/main-web-phpfpm -> <accountid>.dkr.ecr.us-east-1.amazonaws.com/main-web-phpfpm:test-3.18.0-289-gd4737386
Checking cache...
 - <accountid>.dkr.ecr.us-east-1.amazonaws.com/main-web-phpfpm: Not found. Building
Building [<accountid>.dkr.ecr.us-east-1.amazonaws.com/main-web-phpfpm]...
Successfully built 6d3465e2fe20
Successfully tagged <accountid>.dkr.ecr.us-east-1.amazonaws.com/main-web-phpfpm:test-3.18.0-289-gd4737386
The push refers to repository [<accountid>.dkr.ecr.us-east-1.amazonaws.com/main-web-phpfpm]
5c9267607dc1: Pushed
9bc5ddad5e3e: Pushed
2c345fd97596: Pushed
2fdfdeb81619: Pushed
27c6d768710e: Layer already exists
0169fad74b88: Layer already exists
8af3c6bd6621: Layer already exists
1e3ca05d5151: Layer already exists
e7cd8cea3ee0: Layer already exists
862430d50960: Layer already exists
0574259d26c9: Layer already exists
9a7ad28c776d: Layer already exists
a08cf0e71e2a: Layer already exists
cfae40cc952a: Layer already exists
ec21dbbf635f: Layer already exists
f40b92b44d56: Layer already exists
836a2e2925cd: Layer already exists
7fe3caaea95c: Layer already exists
cd1253343b4e: Layer already exists
eab592dd1e81: Layer already exists
21e4fbf5d49d: Layer already exists
708ba1e65ea2: Layer already exists
cb42413394c4: Layer already exists
test-3.18.0-289-gd4737386: digest: sha256:a8267aef45cdfefbda09606cddac17bba1ce1b6ba1e870c4f75d78f7cfdc8d1b size: 5133
There is a new version (1.19.0) of Skaffold available. Download it from:
  https://github.com/GoogleContainerTools/skaffold/releases/tag/v1.19.0

othercorey · 2021-02-17T00:38:07Z

I created a test repository with a repro. However, you have to change the AWS ECR repository to something you have access to.

https://github.com/othercorey/skaffold-5337

This fails to push with --kube-context CLI option and local: push: in skaffold.yaml. My understanding is one of these should trigger the remote push.

skaffold 1.19

$ skaffold-119.exe build --kube-context prod-main -vdebug
time="2021-02-16T18:30:35-06:00" level=info msg="Skaffold &{Version:v1.19.0 ConfigVersion:skaffold/v2beta11 GitVersion: GitCommit:63949e28f40deed44c8f3c793b332191f2ef94e4 GitTreeState:dirty BuildDate:2021-01-28T17:30:48Z GoVersion:go1.14.2 Compiler:gc Platform:windows/amd64}"
time="2021-02-16T18:30:35-06:00" level=debug msg="Update check not enabled, skipping."
time="2021-02-16T18:30:35-06:00" level=info msg="Loaded Skaffold defaults from \"C:\\\\Users\\\\<name>\\\\.skaffold\\\\config\""
time="2021-02-16T18:30:35-06:00" level=debug msg="config version out of date: upgrading to latest \"skaffold/v2beta11\""
time="2021-02-16T18:30:35-06:00" level=debug msg="Defaulting deploy type to kubectl"
time="2021-02-16T18:30:35-06:00" level=info msg="Activated kube-context \"prod-main\""
time="2021-02-16T18:30:35-06:00" level=info msg="Using kubectl context: prod-main"
time="2021-02-16T18:30:35-06:00" level=debug msg="setting Docker user agent to skaffold-v1.19.0"
time="2021-02-16T18:30:35-06:00" level=debug msg="Using builder: local"
time="2021-02-16T18:30:35-06:00" level=debug msg="push value not present, defaulting to false because cluster.PushImages is false"
Generating tags...
 - <accountid>.dkr.ecr.us-east-1.amazonaws.com/skaffold-5337 -> time="2021-02-16T18:30:35-06:00" level=debug msg="Running command: [git describe --tags --always]"
time="2021-02-16T18:30:35-06:00" level=debug msg="Command output: [7dc9da1\n]"
time="2021-02-16T18:30:35-06:00" level=debug msg="Running command: [git status . --porcelain]"
time="2021-02-16T18:30:35-06:00" level=debug msg="Command output: []"
<accountid>.dkr.ecr.us-east-1.amazonaws.com/skaffold-5337:7dc9da1
time="2021-02-16T18:30:35-06:00" level=info msg="Tags generated in 39.9966ms"
Checking cache...
time="2021-02-16T18:30:35-06:00" level=debug msg="Found dependencies for dockerfile: [{test.txt / true}]"
time="2021-02-16T18:30:35-06:00" level=debug msg="push value not present, defaulting to false because cluster.PushImages is false"
 - <accountid>.dkr.ecr.us-east-1.amazonaws.com/skaffold-5337: Found. Tagging
time="2021-02-16T18:30:35-06:00" level=debug msg="push value not present, defaulting to false because cluster.PushImages is false"
time="2021-02-16T18:30:35-06:00" level=info msg="Cache check completed in 28.5141ms"
time="2021-02-16T18:30:35-06:00" level=debug msg="exporting metrics"
time="2021-02-16T18:30:36-06:00" level=debug msg="metrics uploading complete in 760.7898ms"

skaffold 1.16

$ skaffold build --kube-context prod-main -vdebug
time="2021-02-16T18:32:26-06:00" level=info msg="Skaffold &{Version:v1.16.0 ConfigVersion:skaffold/v2beta9 GitVersion: GitCommit:dcdf77cd94ec6a69d116c2123dc8727971e8b14c GitTreeState:clean BuildDate:2020-10-29T01:38:02Z GoVersion:go1.14.2 Compiler:gc Platform:windows/amd64}"
time="2021-02-16T18:32:26-06:00" level=info msg="Loaded Skaffold defaults from \"C:\\\\Users\\\\<name>\\\\.skaffold\\\\config\""
time="2021-02-16T18:32:26-06:00" level=info msg="Activated kube-context \"prod-main\""
time="2021-02-16T18:32:26-06:00" level=debug msg="Defaulting deploy type to kubectl"
time="2021-02-16T18:32:26-06:00" level=info msg="Using kubectl context: prod-main"
time="2021-02-16T18:32:26-06:00" level=debug msg="Using builder: local"
time="2021-02-16T18:32:26-06:00" level=debug msg="setting Docker user agent to skaffold-v1.16.0"
time="2021-02-16T18:32:26-06:00" level=debug msg="push value not present, defaulting to true because localCluster is false"
Generating tags...
 - <accountid>.dkr.ecr.us-east-1.amazonaws.com/skaffold-5337 -> time="2021-02-16T18:32:26-06:00" level=debug msg="Running command: [git describe --tags --always]"
time="2021-02-16T18:32:26-06:00" level=debug msg="Command output: [7dc9da1\n]"
time="2021-02-16T18:32:26-06:00" level=debug msg="Running command: [git status . --porcelain]"
time="2021-02-16T18:32:26-06:00" level=debug msg="Command output: []"
<accountid>.dkr.ecr.us-east-1.amazonaws.com/skaffold-5337:7dc9da1
time="2021-02-16T18:32:26-06:00" level=info msg="Tags generated in 40.0163ms"
Checking cache...
time="2021-02-16T18:32:26-06:00" level=debug msg="Found dependencies for dockerfile: [{test.txt / true}]"
 - <accountid>.dkr.ecr.us-east-1.amazonaws.com/skaffold-5337: Found. Pushing
time="2021-02-16T18:32:26-06:00" level=debug msg="Skipping credential configuration because docker-credential-gcloud is not on PATH."
The push refers to repository [<accountid>.dkr.ecr.us-east-1.amazonaws.com/skaffold-5337]
9a4c892ee8af: Layer already exists
85fcec7ef3ef: Layer already exists
3e5288f7a70f: Layer already exists
56bc37de0858: Layer already exists
1c91bf69a08b: Layer already exists
cb42413394c4: Layer already exists
7dc9da1: digest: sha256:1fefe39e32c7688014ab34e6c4668571486b756d63099c17074d7ff2584c5d88 size: 1569
time="2021-02-16T18:32:28-06:00" level=info msg="Cache check complete in 2.479962s"
There is a new version (1.20.0) of Skaffold available. Download it from:
  https://github.com/GoogleContainerTools/skaffold/releases/tag/v1.20.0

othercorey · 2021-02-24T04:17:00Z

Is there any more debug output I can provide? What triggers the check for a remote repo?

aariacarterweir · 2021-02-24T10:13:49Z

I'm seeing this too on GHCR. I've narrowed it down to using the --kube-context flag to a remote cluster, but having your current context in kubectl set to a local one.

othercorey · 2021-02-25T22:37:15Z

Comparing v1.16.0 to v1.17.0, the only really relevant change I can find is the options refactor:

#4986

nkubala · 2021-02-26T20:00:42Z

i'm investigating this one now to see what changed, our logic to determine whether a cluster is local or not is a bit tangled. while i'm doing that though, @othercorey the section of your skaffold.yaml for specifying push isn't actually doing anything:

local:
  push: <- this is empty, so skaffold interprets it as false

instead, if you change it to

local:
  push: true

it should force a push.

still does seem like something has changed related to cluster detection, so i'll keep looking into it.

othercorey · 2021-02-26T21:31:21Z

@othercorey the section of your skaffold.yaml for specifying push isn't actually doing anything:

Thank you, I will try it out. I think that was probably from following the skaffold.xml docs which omits the false default.

https://skaffold.dev/docs/references/yaml/#build-local-push

Confirmed it does push with local: push: set to true instead default.

If I remove local: and pass a --kube-context, should it still push? It does not.

aariacarterweir · 2021-03-03T00:28:57Z

Interestingly when doing a local build, if your current-context is set to a remote context, it actually runs the push which is consistent with what i found before.

dan-j · 2021-03-08T20:19:58Z

I'm seeing this too on GHCR. I've narrowed it down to using the --kube-context flag to a remote cluster, but having your current context in kubectl set to a local one.

I've just experienced the same issue in the same way that @aariacarterweir posted. Explicitly setting kube context to a remote cluster before running skaffold works as expected.

alewis001 · 2021-03-24T10:10:29Z

I don't know if this helps at all and sorry for the lack of detail on this... I think I've experienced this issue but I'm not sure it's quite what was described here but I think it's possibly related. I found I had to delete my skaffold/config file in order for remote push to work.

My skaffold.yaml has local: {} in the common section and then profiles that (try) to set push to true or false; however, whatever i tried had no effect. The detection of of the cluster didn't seem to work, setting the push attribute in the skaffold.yaml didn't work, etc.

If I ran a skaffold config list it showed a config that had local push to false and that appeared to be overriding everything else. I don't know if that's expected or not but once I deleted the config file, my profile for a remote cluster started working.

tejal29 · 2021-04-12T20:13:01Z

/cc @nkubala to look into this further.

aaron-prindle · 2021-05-10T17:39:37Z

Friendly ping, @nkubala, were you able to get any more information here? To summarize the comments so far:

if the skaffold.yaml has

    local:
        push:

push is set push=false so the behavior is expected in this case. The unexpected case is:

when --kube-context flag is set to a remote cluster but having your current context in kubectl set to a local one, no image is pushed

briandealwis · 2021-05-30T01:50:58Z

@othercorey does this still happen with Skaffold v1.25.0? Following @alewis001's comment, could you please share your ~/.skaffold/config?

gsquared94 · 2021-06-25T11:11:03Z

If I remove local: and pass a --kube-context, should it still push? It does not.

I'm seeing this too on GHCR. I've narrowed it down to using the --kube-context flag to a remote cluster, but having your current context in kubectl set to a local one.

@othercorey, @aariacarterweir this is fixed in #6024

I've created #6078 for the issue: if you specify a remote kubeContext field in the skaffold.yaml file but have current context (or --kubecontext flag) pointing to a local cluster, while also not setting build.push to true, then skaffold incorrectly chooses to not push the built images.

MarlonGamez added kind/bug Something isn't working needs-reproduction needs reproduction from the maintainers to validate the issue is truly a skaffold bug labels Feb 2, 2021

aliwatters mentioned this issue Feb 2, 2021

Microk8s - Error: container <container-name> is waiting to start: <image with missing tag> can't be pulled #5327

Open

othercorey changed the title ~~{1.17+] Build does not push images to remote repository~~ [1.17+] Build does not push images to remote repository Feb 4, 2021

MarlonGamez added the triage/discuss Items for discussion label Feb 8, 2021

nkubala added priority/p2 May take a couple of releases help wanted We would love to have this done, but don't have the bandwidth, need help from contributors labels Feb 12, 2021

MarlonGamez removed the triage/discuss Items for discussion label Mar 8, 2021

tejal29 added this to the Sustainability and Tech Debt (v1.27.0) milestone May 17, 2021

tejal29 added the planning/Q2-21 label May 18, 2021

briandealwis removed the priority/p2 May take a couple of releases label May 31, 2021

gsquared94 added the priority/p2 May take a couple of releases label Jun 16, 2021

gsquared94 self-assigned this Jun 16, 2021

gsquared94 mentioned this issue Jun 16, 2021

Fix setting kubeContext in skaffold #6024

Merged

gsquared94 removed help wanted We would love to have this done, but don't have the bandwidth, need help from contributors needs-reproduction needs reproduction from the maintainers to validate the issue is truly a skaffold bug labels Jun 21, 2021

gsquared94 mentioned this issue Jun 21, 2021

delete #5176

Closed

gsquared94 closed this as completed in #6024 Jun 24, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[1.17+] Build does not push images to remote repository #5334

[1.17+] Build does not push images to remote repository #5334

othercorey commented Feb 2, 2021 •

edited

Loading

MarlonGamez commented Feb 2, 2021 •

edited

Loading

othercorey commented Feb 4, 2021 •

edited

Loading

MarlonGamez commented Feb 8, 2021

othercorey commented Feb 10, 2021

othercorey commented Feb 17, 2021

othercorey commented Feb 24, 2021

aariacarterweir commented Feb 24, 2021

othercorey commented Feb 25, 2021

nkubala commented Feb 26, 2021

othercorey commented Feb 26, 2021 •

edited

Loading

aariacarterweir commented Mar 3, 2021

dan-j commented Mar 8, 2021 •

edited

Loading

alewis001 commented Mar 24, 2021

tejal29 commented Apr 12, 2021

aaron-prindle commented May 10, 2021 •

edited

Loading

briandealwis commented May 30, 2021

gsquared94 commented Jun 25, 2021

[1.17+] Build does not push images to remote repository #5334

[1.17+] Build does not push images to remote repository #5334

Comments

othercorey commented Feb 2, 2021 • edited Loading

Expected behavior

Actual behavior

Information

Steps to reproduce the behavior

MarlonGamez commented Feb 2, 2021 • edited Loading

othercorey commented Feb 4, 2021 • edited Loading

MarlonGamez commented Feb 8, 2021

othercorey commented Feb 10, 2021

othercorey commented Feb 17, 2021

skaffold 1.19

skaffold 1.16

othercorey commented Feb 24, 2021

aariacarterweir commented Feb 24, 2021

othercorey commented Feb 25, 2021

nkubala commented Feb 26, 2021

othercorey commented Feb 26, 2021 • edited Loading

aariacarterweir commented Mar 3, 2021

dan-j commented Mar 8, 2021 • edited Loading

alewis001 commented Mar 24, 2021

tejal29 commented Apr 12, 2021

aaron-prindle commented May 10, 2021 • edited Loading

briandealwis commented May 30, 2021

gsquared94 commented Jun 25, 2021

othercorey commented Feb 2, 2021 •

edited

Loading

MarlonGamez commented Feb 2, 2021 •

edited

Loading

othercorey commented Feb 4, 2021 •

edited

Loading

othercorey commented Feb 26, 2021 •

edited

Loading

dan-j commented Mar 8, 2021 •

edited

Loading

aaron-prindle commented May 10, 2021 •

edited

Loading