Skipping deploy due to sync error: copying files: didn't sync any files

[alexgalkin]

Expected behavior

Files should be synced to running container

Actual behavior

Files are not updated; receiving "Skipping deploy due to sync error: copying files: didn't sync any files" warning message

Information

• Skaffold version: v1.3.1
• Operating system: macOS Catalina 10.15.3 (the latest)
• Contents of skaffold.yaml:

apiVersion: skaffold/v2alpha3
kind: Config
metadata:
  name: hello-world
build:
  artifacts:
  - image: ananthrs/hello-k8s
    context: .
    sync:
      infer:
      - '**/*.js'
  local:
    push: false
  tagPolicy:
    envTemplate:
      template: "{{.IMAGE_NAME}}:latest"
deploy:
  kubeContext: minikube
  helm:
    releases:
    - name: hello-world-release
      chartPath: helm/hello-k8s
      namespace: demo
      setValueTemplates:
        image.tag: "latest"
      recreatePods: true

Steps to reproduce the behavior

1. https://github.com/alexgalkin/k8s-demo the whole project for testing is here
2. skaffold dev --port-forward -v debug

Note: I have also switched context to "minikube"

What I actually see in the console:

Watching for changes...
DEBU[0043] Change detected notify.Write: "/Users/alexandergalkin/projects/ptv/alexgalkin/k8s-demo/src/server.js" 
DEBU[0044] Found dependencies for dockerfile: [{src/package-lock.json /usr/src/app true} {src/package.json /usr/src/app true} {src/server.js /usr/src/app false}] 
INFO[0044] files modified: [src/server.js]              
DEBU[0044] Found dependencies for dockerfile: [{src/package-lock.json /usr/src/app true} {src/package.json /usr/src/app true} {src/server.js /usr/src/app false}] 
Syncing 1 files for ananthrs/hello-k8s:001835f39386796c0623046c9a434a5902ff04d1cd4b02c9447d99c0a0380985
INFO[0044] Copying files: map[src/server.js:[/usr/src/app]] to ananthrs/hello-k8s:001835f39386796c0623046c9a434a5902ff04d1cd4b02c9447d99c0a0380985 
DEBU[0044] getting client config for kubeContext: `minikube` 
WARN[0044] Skipping deploy due to sync error: copying files: didn't sync any files 
Watching for changes...

[alexgalkin]

extra notes:

kubectl version: v1.15.6
helm version: v2.14.3
minikube version: v1.7.2

[kvokka]

Got the same with Skaffold 1.4.0

[alexgalkin]

Used the latest build.

 ./skaffold version                    
c96b1f

The issue is still there.

Watching for changes...
DEBU[0142] Change detected notify.Write: "/Users/alexandergalkin/projects/ptv/k8s-demo/src/server.js" 
DEBU[0143] Found dependencies for dockerfile: [{src/package-lock.json /usr/src/app true} {src/package.json /usr/src/app true} {src/server.js /usr/src/app false}] 
INFO[0143] files modified: [src/server.js]              
DEBU[0143] Found dependencies for dockerfile: [{src/package-lock.json /usr/src/app true} {src/package.json /usr/src/app true} {src/server.js /usr/src/app false}] 
Syncing 1 files for ananthrs/hello-k8s:2368470e4f9fc1f366e24eb57b9bae71204b42bcbc00779b287684b4b9761bfb
INFO[0143] Copying files: map[src/server.js:[/usr/src/app]] to ananthrs/hello-k8s:2368470e4f9fc1f366e24eb57b9bae71204b42bcbc00779b287684b4b9761bfb 
DEBU[0143] getting client config for kubeContext: `minikube` 
WARN[0143] Skipping deploy due to sync error: copying files: didn't sync any files 
Watching for changes.

[vincentschut]

Same problem over here. Downgrading skaffold to 1.2.0 helps for the time being.

[pre]

I have been experiencing problems with Skaffold Sync (v1.4.0). I found out the namespace in $KUBECONFIG affects Skaffold's ability to sync files. Skaffold sync will only work when the Kubernetes namespace in $KUBECONFIG, at the time of starting Skaffold, matches the namespace of the artifact (application) being deployed.

See #1668 (comment) for details

[nkubala]

@pre thanks for the investigation, and sorry there hasn't been any movement here. this does indeed look like a namespacing issue based on your repro's, but unfortunately we haven't had any time on the team to investigate it.

would you be able to provide a small sample project along with a kubeconfig that we can use to repro?

also as a side note, we might want to consider handling this more gracefully in skaffold, i.e. continuing with the deploy even when we hit a sync error.

[alexgalkin]

@nkubala there is a sample project mentioned in my very first comment on this issue.

[dgageot]

@alexgalkin I ran your sample and saw those warnings:

WARN[0000] image [ananthrs/hello-k8s:9a585910e9b9eb9b746a538cd73ede89c602c515c158f774f93ef89dc86d6dce] is not used.
WARN[0000] image [ananthrs/hello-k8s] is used instead.
WARN[0000] See helm sample for how to replace image names with their actual tags: https://github.com/GoogleContainerTools/skaffold/blob/master/examples/helm-deployment/skaffold.yaml
WARN[0001] error adding label to runtime object: patching resource default/"hello-world-release-k8s-demo-test-connection": pods "hello-world-release-k8s-demo-test-connection" not found

Can you try to fix them and tell us if that fixes the sync?

[tstromberg]

Closing because we haven't heard back. Please comment when you get a chance to update this.

[pre]

@tstromberg the issue still exists. Here’s instructions for reproducing it: #1668 (comment)

That other issue was closed earlier for some reason but the problem is the same.

I can open a new issue if that’s needed but i think all the required info is in #1668 (comment)

[joekendal]

Console

WARN[2162] Skipping deploy due to sync error: copying files: running [kubectl --context gke_ticketing-dev-283015_europe-west2-a_ticketing-dev exec auth-depl-58cf6875b5-qb95n --namespace default -c auth -i -- tar xmf - -C / --no-same-owner]
 - stdout: ""
 - stderr: "Error from server: error dialing backend: ssh: rejected: connect failed (Connection refused)\n"
 - cause: exit status 1 

Config

apiVersion: skaffold/v2alpha3
kind: Config
deploy:
  kubectl:
    manifests:
      - ./infra/k8s/*
build:
  # local:
  #   push: false
  googleCloudBuild:
    projectId: ticketing-dev-283015
  artifacts:
    - image: eu.gcr.io/ticketing-dev-283015/auth
      context: auth
      docker:
        dockerfile: Dockerfile
      sync:
        manual:
          - src: 'src/**/*.ts'
            dest: .

[joekendal]

Console

WARN[2162] Skipping deploy due to sync error: copying files: running [kubectl --context gke_ticketing-dev-283015_europe-west2-a_ticketing-dev exec auth-depl-58cf6875b5-qb95n --namespace default -c auth -i -- tar xmf - -C / --no-same-owner]
 - stdout: ""
 - stderr: "Error from server: error dialing backend: ssh: rejected: connect failed (Connection refused)\n"
 - cause: exit status 1 

This was resolved with:

$ skaffold delete
$ skaffold dev

[tejal29]

From the error logs mentioned by @joekendal looks like this is possible dupe of #4966. I am going to close this issue since we tracking it there.

[maurerbot]

I've having the same issue...

kubectl ctx
microk8s

skaffold dev --port-forward --default-repo localhost:32000 --kube-context='microk8s'

INFO[0042] files modified: [src/app/src/app.py]
Syncing 1 files for localhost:32000/nbninja_app:latest@sha256:05095f1f3aa6f91561a4a64f665aab95bce7f1ed94d98eecda4ecc25942b81e0
INFO[0042] Copying files: map[src/app/src/app.py:[/app/src/app.py]] to localhost:32000/nbninja_app:latest@sha256:05095f1f3aa6f91561a4a64f665aab95bce7f1ed94d98eecda4ecc25942b81e0
DEBU[0042] getting client config for kubeContext: `microk8s`
WARN[0042] Skipping deploy due to sync error: copying files: didn't sync any files

 apiVersion: skaffold/v2beta13
 50 kind: Config
 51
 52 ## BUILD
 53
 54 build:
 55   insecureRegistries:
 56     # Microk8s local registry. If not enabled run:
 57     # $ microk8s enable registry
 58   - localhost:32000
 59
 60   artifacts:
 73     - image: nbninja_app
 74       context: src/app
 75       sync:
 76         manual:
 77           - src: 'src/**/*'
 78             dest: '/app'
 79         # infer:
 80           # - 'src/app/src/app.py'
 81           # - 'src/app/src/db/*.py'
 82
 83   tagPolicy:
 84     sha256: {}
 85
 86   local:
 87     push: true
 88     useBuildkit: true

[nawarnoori]

Exact same issue here, was on 1.35, downgraded to 1.20 and it's working now.

Not sure which version "broke" it, but for me at least I can confirm it's worth trying out 1.20 to see if it fixes it for you.