Dev build cleanup deletes all layers

[tdickman]

Expected behavior

Skaffold dev should only cleanup layers it created.

Actual behavior

Skaffold dev cleans up all layers for a given image (even layers that already existed before it started).

For example, if I run skaffold run once, then run skaffold dev to start developing my application, and later hit control-c to stop the process, skaffold deletes all layers of the image it used, even the layers it didn't build (ex: layers that were built during skaffold run).

My current workflow is to run skaffold dev to develop my application, and use control-c when I am done working on it for the time being. I then resume with skaffold dev again the next day. This used to start the application in a few seconds (since the previous layers already existed). Now it results in 5 - 10 minutes of build time since it has to rebuild a multi-gigabit docker image.

I believe we should either:

• Update skaffold to only delete new layers that it is responsible for
• Disable image cleanup (pruning) as a default

Information

• Skaffold version: v0.28.0
• Operating system: Ubuntu 18.10
• Contents of skaffold.yaml:

apiVersion: skaffold/v1beta7
kind: Config
build:
  artifacts:
    - image: localhost:32000/app
deploy:
  helm:
    releases:
      - name: app
        chartPath: infrastructure/kube
        namespace: default
        setValueTemplates:
          image.repository: "{{.IMAGE_NAME}}"
          image.tag: "{{.DIGEST}}"
        imageStrategy:
          helm: {}
        valuesFiles:
          - infrastructure/kube/values.yaml

Steps to reproduce the behavior

1. Run skaffold run once (to build and deploy an app)
2. Run skaffold dev
3. Control-c out of skaffold dev
4. Run skaffold dev again

[tejal29]

The quick fix for you would be to use --no-prune flag on your command line.
You can also set environment variable
"SKAFFOLD_NO_PRUNE=false"

I was digging in to the cleanup code and i think that is the intended behaviour.
We delete images which are built by skaffold. So all layers mentioned in your global skaffold config will be deleted.
If you want to iterate a specific layer, then i would suggest add a skaffold.yaml for the layer and run skaffold dev on this skaffold.yaml.

e.g.

root/
skaffold.yaml
layer1/
  src/
  test/
layer2/
  src/
app/
  skaffold.yaml
  src/

So your workflow would be:

1. Build and deploy once
cd root
skaffold run 
2. Iterate over a specific layer.
skaffold dev -f app/skaffold.dev

Does that help?

[nkubala]

skaffold does its pruning through the Force and PruneChildren options passed to the docker client call (or the --force-rm flag passed to docker build if building through the CLI). I think the majority of users iterating with skaffold dev don't want to see dozens or even hundreds of images stacking up in their docker daemon when they're done using skaffold, though I could be wrong.

I don't think we want to turn this off by default, but we could expose pruning options through the config - for your specific case, setting PruneChildren to false should fix your issue. it's not that skaffold is removing unrelated layers, but since the previously built layers are being reused in images build through your dev loop, they get cleaned up at the end.

would you be interested in sending a PR for this?

[charlyx]

@nkubala I'm interested in sending a PR, if that's OK 😄

[priyawadhwa]

@charlyx that would be great!

[nkubala]

closed through #2113