When using extensions, if the last layer in the run image is duplicated, then the process fail with an error

[juancoen]

Summary

When using extensions, if the run image latest layer is duplicated, then the process fails with an error like:

[extender (run)] ERROR: failed to extend run image: failed to copy extended image to output directory: open /kaniko/cache/base/sha256:f07ab0840dba11b097245ed458798aae52e6f5616f153e62825ca3ad4a7d3c69/blobs/sha256/b4af3bc866d42dd85a3862570db7d79172e169904d0def439abeae32a00dea16: no such file or directory

---

Reproduction

Steps

1. Create a run image which latest layer is duplicated, for example:

FROM ...

RUN command

WORKDIR /usr/src/app

RUN another command

WORKDIR /usr/src/app

2. Use the image as a builder stack run-image
3. Use an extension that adds something to the run-image
4. Try to create an image using this builder

Current behavior

The build fails with this error:

[extender (run)] ERROR: failed to extend run image: failed to copy extended image to output directory: open /kaniko/cache/base/sha256:f07ab0840dba11b097245ed458798aae52e6f5616f153e62825ca3ad4a7d3c69/blobs/sha256/b4af3bc866d42dd85a3862570db7d79172e169904d0def439abeae32a00dea16: no such file or directory

Expected behavior

The build must complete successfully

---

Context

lifecycle version

0.18.4

platform version(s)

pack report                                                                                                
Pack:
  Version:  0.32.1+git-b14250b.build-5241
  OS/Arch:  darwin/amd64

Default Lifecycle Version:  0.17.2

Supported Platform APIs:  0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.10, 0.11, 0.12

Config:
  experimental = true
  layout-repo-dir = "/Users/jcoen/.pack/layout-repo"

Client:
 Version:    25.0.3
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1-desktop.4
    Path:     /Users/jcoen/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.5-desktop.1
    Path:     /Users/jcoen/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.24
    Path:     /Users/jcoen/.docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/jcoen/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.21
    Path:     /Users/jcoen/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     /Users/jcoen/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.0.0
    Path:     /Users/jcoen/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/jcoen/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.4.1
    Path:     /Users/jcoen/.docker/cli-plugins/docker-scout
WARNING: Plugin "/Users/jcoen/.docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /Users/jcoen/.docker/cli-plugins/docker-scan: no such file or directory

Server:
 Containers: 16
  Running: 2
  Paused: 0
  Stopped: 14
 Images: 40
 Server Version: 25.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.12-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 10
 Total Memory: 62.69GiB
 Name: docker-desktop
 ID: 0e22bbb1-6f60-4587-b026-889ca9a4c6c5
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

anything else?

no

[jericop]

I'm not sure if this is related, but I'm getting a similar error when trying to follow the buildpacks.io guide to create an extension here.

Running on an amd64 mac.
pack version: 0.33.2+git-f2cffc4.build-5562
lifecycle version: 0.18.5
I also tried it with lifecycle versions 0.18.1, 0.18.2, 0.18.3, and 0.18.4. All failed with the same error.

➜  buildpacks pack --version
0.33.2+git-f2cffc4.build-5562
➜  buildpacks 
➜  buildpacks docker run -d --rm -p 5000:5000 --name registry registry:2
9d9cc6bb0fafdd095d76d8ca9767ce8b9a4102cf502b36da6198ab6cf913a781
➜  buildpacks pack builder create localhost:5000/extensions-builder \  
  --config $PWD/samples/builders/alpine/builder.toml \
  --publish
Successfully created builder image localhost:5000/extensions-builder
Tip: Run pack build <image-name> --builder localhost:5000/extensions-builder to use this builder
➜  buildpacks docker buildx build \                                  
  --file $PWD/samples/base-images/alpine/run/curl.Dockerfile \
  --tag localhost:5000/run-image-curl --push .
[+] Building 0.7s (8/8) FINISHED                                                                                                                               docker:desktop-linux
 => [internal] load build definition from curl.Dockerfile                                                                                                                      0.0s
 => => transferring dockerfile: 631B                                                                                                                                           0.0s
 => [internal] load metadata for docker.io/curlimages/curl:latest                                                                                                              0.0s
 => [internal] load .dockerignore                                                                                                                                              0.0s
 => => transferring context: 2B                                                                                                                                                0.0s
 => [1/3] FROM docker.io/curlimages/curl:latest                                                                                                                                0.0s
 => CACHED [2/3] RUN curl --version                                                                                                                                            0.0s
 => CACHED [3/3] RUN addgroup -g 1001 cnb &&   adduser -u 1000 -G cnb -s /bin/bash -D cnb                                                                                      0.0s
 => exporting to image                                                                                                                                                         0.0s
 => => exporting layers                                                                                                                                                        0.0s
 => => writing image sha256:d494e6196217dba06eed7432778d5ca892412d7da8cb7712738790c634d9fc93                                                                                   0.0s
 => => naming to localhost:5000/run-image-curl                                                                                                                                 0.0s
 => pushing localhost:5000/run-image-curl with docker                                                                                                                          0.5s
 => => pushing layer 3291863abd2f                                                                                                                                              0.1s
 => => pushing layer e2be6162e64d                                                                                                                                              0.1s
 => => pushing layer 8e406d5548c1                                                                                                                                              0.1s
 => => pushing layer b921b7bdd945                                                                                                                                              0.1s
 => => pushing layer 6094041ec9a7                                                                                                                                              0.1s
 => => pushing layer d4629ddd9445                                                                                                                                              0.1s
 => => pushing layer 606b0fac4a3a                                                                                                                                              0.2s
 => => pushing layer f5a6f6774bce                                                                                                                                              0.2s
 => => pushing layer f83640e95477                                                                                                                                              0.1s
 => => pushing layer 0ea665f430c2                                                                                                                                              0.1s
 => => pushing layer fe12ac6dc06c                                                                                                                                              0.3s
 => => pushing layer 221327e0f59c                                                                                                                                              0.4s
 => => pushing layer 994393dc58e7                                                                                                                                              0.5s
➜  buildpacks pack build hello-extensions \                             
  --builder localhost:5000/extensions-builder \
  --env BP_EXT_DEMO=1 \
  --env BP_REQUIRES=vim,curl \
  --path $PWD/samples/apps/java-maven \
  --pull-policy always \
  --network host \
  --verbose
Builder localhost:5000/extensions-builder is untrusted
As a result, the phases of the lifecycle which require root access will be run in separate trusted ephemeral containers.
For more information, see https://medium.com/buildpacks/faster-more-secure-builds-with-pack-0-11-0-4d0c633ca619
Pulling image localhost:5000/extensions-builder:latest
latest: Pulling from extensions-builder
4abcf2066143: Already exists 
156c2a1e46a7: Already exists 
a86538552170: Already exists 
1f0346ad010f: Already exists 
f9c8ac9ab409: Already exists 
f3fc0f991d83: Already exists 
f10bc4ceffe5: Already exists 
569fe741231d: Already exists 
6d5077fd4ab0: Already exists 
b7237553612e: Already exists 
1c971575c8d6: Already exists 
4da96f148d45: Already exists 
90903455c218: Already exists 
62b7c2336dfd: Already exists 
9e90005d9e74: Already exists 
c983951b913c: Already exists 
4b2fe7e481b4: Already exists 
32d2a27e557e: Already exists 
25f51db0314e: Already exists 
89732bc75041: Already exists 
Digest: sha256:042ee2582ffa1fbddf5bf4e04bacb73084742e51216905ce7ebbe7c6371b948b
Status: Downloaded newer image for localhost:5000/extensions-builder:latest
CheckReadAccess succeeded for the run image cnbs/sample-base-run:alpine
Selected run image cnbs/sample-base-run:alpine
Pulling image cnbs/sample-base-run:alpine
alpine: Pulling from cnbs/sample-base-run
Digest: sha256:3d714eab0232f25bf14f3c5e8d15a4f813865728a5833d9cd9804d54928f871c
Status: Image is up to date for cnbs/sample-base-run:alpine
Pulling image buildpacksio/lifecycle:0.18.5
0.18.5: Pulling from buildpacksio/lifecycle
Digest: sha256:16a4c9e945381ad182fbb92710f0643e1c434ce11c3c43fb0ce6fcfce0d5efb7
Status: Image is up to date for buildpacksio/lifecycle:0.18.5
Creating builder with the following buildpacks:
-> samples/java-maven@0.0.2
-> samples/kotlin-gradle@0.0.2
-> samples/hello-universe@0.0.1
-> samples/hello-moon@0.0.1
-> samples/hello-world@0.0.1
-> samples/hello-extensions@0.0.1
Provided Environment Variables
  BP_EXT_DEMO=1
  BP_REQUIRES=vim,curl
Using build cache volume pack-cache-library_hello-extensions_latest-605c1c945967.build
===> ANALYZING
Running the analyzer on OS linux from image buildpacksio/lifecycle:0.18.5 with:
Container Settings:
  Args: /cnb/lifecycle/analyzer -log-level debug -daemon -run /layers/run.toml -run-image cnbs/sample-base-run:alpine -launch-cache /launch-cache hello-extensions
  System Envs: CNB_USER_ID=1000 CNB_GROUP_ID=1001 CNB_PLATFORM_API=0.12
  Image: buildpacksio/lifecycle:0.18.5
  User: root
  Labels: map[author:pack]
Host Settings:
  Binds: /var/run/docker.sock:/var/run/docker.sock pack-cache-library_hello-extensions_latest-605c1c945967.launch:/launch-cache pack-layers-sybjbqfvld:/layers pack-app-dygkglcczv:/workspace
  Network Mode: host
[analyzer] Starting analyzer...
[analyzer] Parsing inputs...
[analyzer] Ensuring privileges...
[analyzer] Executing command...
[analyzer] Timer: Analyzer started at 2024-02-25T02:40:59Z
[analyzer] Found image with identifier "dbbad154b2ca1fe6de578774bb3ada003774a0fa55083ebfc6e5c29013124c5b"
[analyzer] Restoring data for SBOM from previous image
[analyzer] Retrieving previous image SBOM layer for "sha256:e45a9ce4189efaae8efbeb9ea4b634f225ded1a4b5711e7c0be6a2c97b457ad3"
[analyzer] Found image with identifier "006ca3744686da3362e56319682d0fd1f221dd9554a11424f1215e499b1555a6"
[analyzer] Timer: Analyzer ran for 1.029356ms and ended at 2024-02-25T02:40:59Z
[analyzer] Run image info in analyzed metadata is: 
[analyzer] {"Reference":"006ca3744686da3362e56319682d0fd1f221dd9554a11424f1215e499b1555a6","Image":"cnbs/sample-base-run:alpine","Extend":false,"target":{"os":"linux","arch":"amd64"}}
===> DETECTING
Running the detector on OS linux from image pack.local/builder/6561706c71706363636f:latest with:
Container Settings:
  Args: /cnb/lifecycle/detector -app /workspace -log-level debug
  System Envs: CNB_EXPERIMENTAL_MODE=warn CNB_PLATFORM_API=0.12
  Image: pack.local/builder/6561706c71706363636f:latest
  User: 
  Labels: map[author:pack]
Host Settings:
  Binds: pack-layers-sybjbqfvld:/layers pack-app-dygkglcczv:/workspace
  Network Mode: host
[detector] Starting detector...
[detector] Parsing inputs...
[detector] Ensuring privileges...
[detector] Executing command...
[detector] Warning: Platform requested experimental feature 'Dockerfiles'
[detector] Timer: Detector started at 2024-02-25T02:40:59Z
[detector] Checking for match against descriptor: {linux   []}
[detector] Checking for match against descriptor: {linux   []}
[detector] Checking for match against descriptor: {linux   []}
[detector] Checking for match against descriptor: {linux   []}
[detector] ======== Results ========
[detector] pass: samples/vim@0.0.1
[detector] pass: samples/curl@0.0.1
[detector] pass: samples/cowsay@0.0.1
[detector] pass: samples/hello-extensions@0.0.1
[detector] Resolving plan... (try #1)
[detector] skip: samples/cowsay@0.0.1 provides unused cowsay
[detector] 3 of 4 buildpacks participating
[detector] samples/vim              0.0.1
[detector] samples/curl             0.0.1
[detector] samples/hello-extensions 0.0.1
[detector] Timer: Detector ran for 16.136781ms and ended at 2024-02-25T02:40:59Z
[detector] Timer: Generator started at 2024-02-25T02:40:59Z
[detector] Running generate for extension samples/vim@0.0.1
[detector] Looking up extension
[detector] Finding plan
[detector] Invoking command
[detector] Creating plan directory
[detector] Preparing paths
[detector] Running generate command
[detector] Reading output files
[detector] Found '1' Dockerfiles for processing
[detector] Finished running generate for extension samples/vim@0.0.1
[detector] Running generate for extension samples/curl@0.0.1
[detector] Looking up extension
[detector] Finding plan
[detector] Invoking command
[detector] Creating plan directory
[detector] Preparing paths
[detector] Running generate command
[detector] Reading output files
[detector] Found '1' Dockerfiles for processing
[detector] Finished running generate for extension samples/curl@0.0.1
[detector] Checking run image
[detector] Found a run.Dockerfile from extension 'samples/curl' setting run image to 'localhost:5000/run-image-curl' 
[detector] Warning: new runtime base image 'localhost:5000/run-image-curl' not found in run metadata
[detector] Updating analyzed metadata with new run image 'localhost:5000/run-image-curl'
[detector] Copying Dockerfiles
[detector] Copying /tmp/cnb-extensions-generated.1096228511/samples_vim/build.Dockerfile to /layers/generated/build/samples_vim/Dockerfile
[detector] Copying /tmp/cnb-extensions-generated.1096228511/samples_curl/run.Dockerfile to /layers/generated/run/samples_curl/Dockerfile
[detector] Timer: Generator ran for 11.333892ms and ended at 2024-02-25T02:40:59Z
[detector] Run image info in analyzed metadata is: 
[detector] {"Reference":"localhost:5000/run-image-curl","Image":"localhost:5000/run-image-curl","Extend":false}
Pulling image localhost:5000/run-image-curl
latest: Pulling from run-image-curl
Digest: sha256:9053673c6b256b37e95bfa1a5e65622a97d4a59187c8f6187410659f75d4b011
Status: Image is up to date for localhost:5000/run-image-curl:latest
ERROR: failed to build: executing lifecycle: open /var/folders/sd/l4j9q6l96bngknrk2xvdhjyh0000gq/T/extend-run-image-scratch778994993/blobs/sha256.tar: no such file or directory
➜  buildpacks 

[natalieparellano]

Thinking about this further - we can fix this in the lifecycle, but I wonder if it makes sense to error (or at least warn) when the run image contains a duplicate top layer, because the resulting app image will not be rebased correctly. Looking here, we consider all layers underneath (and including) the first instance of the top layer diff ID to be the base image. The alternative would be to change how we do this in imgutil.

@jabrown85 do you have any thoughts here?

[natalieparellano]

@jericop thanks for this - I'm not able to reproduce 😕 - it does look like a different issue as I believe the error message is being emitted by pack. Is it possible you could open an issue on the pack repo?

[driv]

@jericop were you able to execute that step of the tutorial?