ADD --checksum=sha512 uses sha256 algorithm to compare digest

[hoylemd]

Description

When including a checksum in an ADD operation in a Dockerfile, the specified algorithm doesn't appear to be respected, despite the documentation stating that other checksum algorithms are supported.

For example, if I have the following instruction in a Dockerfile:

ADD --checksum=sha512:3d425c5a102d441da33030949ba5ec22e388ed0529c298a1984d62486d4924806949708b834229206ee5a36ba30f6de6d09989019e5790a8b665539f9489efd5 \
    https://github.com/ArtifexSoftware/ghostpdl-downloads/releases/download/gs10040/ghostscript-10.04.0.tar.gz ghostscript-10.04.0.tar.gz

and run docker build . in that Dockerfile's directory, I get this error:

 => ERROR [3/9] ADD --checksum=sha512:3d425c5a102d441da33030949ba5ec22e388ed0529c298a1984d62486d4924806949708b834229206ee5a36ba30f6de6d09989019e5790a8b665539f9489efd5     https://github.com/ArtifexSoftware/ghostpdl-downloads/releases/download/gs10040/ghostscript-10.04.0.tar.gz ghostscript-10.04.0.tar.gz                           33.8s
------
 > [3/9] ADD --checksum=sha512:3d425c5a102d441da33030949ba5ec22e388ed0529c298a1984d62486d4924806949708b834229206ee5a36ba30f6de6d09989019e5790a8b665539f9489efd5     https://github.com/ArtifexSoftware/ghostpdl-downloads/releases/download/gs10040/ghostscript-10.04.0.tar.gz ghostscript-10.04.0.tar.gz:
------
ERROR: failed to solve: digest mismatch sha256:c764dfbb7b13fc71a7a05c634e014f9bb1fb83b899fe39efc0b6c3522a9998b1: sha512:3d425c5a102d441da33030949ba5ec22e388ed0529c298a1984d62486d4924806949708b834229206ee5a36ba30f6de6d09989019e5790a8b665539f9489efd5

On the last line, note that it says:

digest mismatch sha256:c76...

but the --checksum argument in the dockerfile is

--checksum=sha512:3d4...

So docker doesn't appear to be respecting the specified checksum algorithm when specified in an ADD step.

Reproduce

docker build .

Expected behavior

Docker should use the specified checksum algorithm (e.g. sha512) to validate the added file.

docker version

Client:
 Version:           27.3.1
 API version:       1.47
 Go version:        go1.22.7
 Git commit:        ce12230
 Built:             Fri Sep 20 11:38:18 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.36.0 (175267)
 Engine:
  Version:          27.3.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.7
  Git commit:       41ca978
  Built:            Fri Sep 20 11:41:19 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.21
  GitCommit:        472731909fa34bd7bc9c087e4c27943f9835f111
 runc:
  Version:          1.1.13
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    27.3.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  ai: Ask Gordon - Docker Agent (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/michaelhoyle/.docker/cli-plugins/docker-ai
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.18.0-desktop.2
    Path:     /Users/michaelhoyle/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.30.3-desktop.1
    Path:     /Users/michaelhoyle/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.37
    Path:     /Users/michaelhoyle/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.15
    Path:     /Users/michaelhoyle/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/michaelhoyle/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.27
    Path:     /Users/michaelhoyle/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/michaelhoyle/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     /Users/michaelhoyle/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/michaelhoyle/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.15.0
    Path:     /Users/michaelhoyle/.docker/cli-plugins/docker-scout
WARNING: Plugin "/Users/michaelhoyle/.docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /Users/michaelhoyle/.docker/cli-plugins/docker-scan: no such file or directory

Server:
 Containers: 5
  Running: 0
  Paused: 0
  Stopped: 5
 Images: 36
 Server Version: 27.3.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.14-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 10
 Total Memory: 7.654GiB
 Name: docker-desktop
 ID: a26e5477-a5f4-4cf4-98b9-bb2ce55d2c8f
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/michaelhoyle/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Additional Info

Example dockerfile to reproduce the bug:

FROM python:3.11

# Build gs
WORKDIR /root
ADD --checksum=sha512:3d425c5a102d441da33030949ba5ec22e388ed0529c298a1984d62486d4924806949708b834229206ee5a36ba30f6de6d09989019e5790a8b665539f9489efd5 \
    https://github.com/ArtifexSoftware/ghostpdl-downloads/releases/download/gs10040/ghostscript-10.04.0.tar.gz ghostscript-10.04.0.tar.gz
RUN tar -zxvf ghostscript-10.04.0.tar.gz
WORKDIR ./ghostscript-10.04.0
RUN ./configure
RUN make -j 16
RUN make install
RUN cd .. && rm ghostscript-10.04.0.tar.gz

the checksum for the tarball can be validated here: https://www.ghostscript.com/releases/gsdnld.html

[hoylemd]

I originally asked about this on StackOverflow, thinking I was just doing something wrong, but someone else was able to reproduce the issue. They also supplied a workaround, in case anyone finds this and needs it.

[thaJeztah]

Looks like an issue in BuildKit or the Dockerfile frontend. Let me transfer this to the buildx repository, which is closer related, and handled by the team working on build

[tonistiigi]

@dvdksn Where does this originate? I don't think this is supported https://github.com/moby/buildkit/blob/v0.17.2/source/http/source.go#L407 and in the implementation PR I can see my comment explaining why other algorithms are not possible moby/buildkit#3093 (comment)

[dvdksn]

@tonistiigi hmm looks like I added it in moby/buildkit#5237, but I can't recall what src I used to conjure that information up. I'll update it.

[thaJeztah]

It's confusing though, because sha384, and sha512 don't produce an error immediately; so they seem to be supported until it fails with an obscure error, so it seems some parts of the code accept it (probably because those algorithms are registered in go-digest), but other parts do not;

sha256 works (as expected)

#6 [2/3] ADD --checksum=sha256:f28da2f67cafc5186812966ea4a17ac54b57a9578070aec019e3e297d1aee532 https://get.docker.com out/install.sh
#6 DONE 0.0s

#7 [3/3] RUN ls -l /out
#7 0.254 total 24
#7 0.254 -rw-------    1 root     root         22115 Nov  8 11:06 install.sh
#7 DONE 0.3s

md5 fails with a somewhat useful error (unsupported digest algorithm), although it would be good for it to tell what algorithm is unsupported (md5), and what algorithms are supported (sha256 only currently);

docker build --no-cache --progress=plain -<<'EOF'
FROM busybox
ADD --checksum=md5:6d94d4a82b6fee20c73acb7eb6f71613 https://get.docker.com out/install3.sh
RUN ls -l /out
EOF

#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s
Dockerfile:2
--------------------
   1 |     FROM busybox
   2 | >>> ADD --checksum=md5:6d94d4a82b6fee20c73acb7eb6f71613 https://get.docker.com out/install3.sh
   3 |     RUN ls -l /out
   4 |
--------------------
ERROR: failed to solve: unsupported digest algorithm

But sha512 and sha384 fail with a much more obscure error; the dockerfile parsing doesn't complain, but it only fails once it actualy is tried;

docker build --no-cache --progress=plain -<<'EOF'
FROM busybox
ADD --checksum=sha512:c6337ff13c9f58c2016cccf88ace46660b22ade0931b424b4a05d009770782894710e9c61aacd8cd76c53fcb9f5fe42b39c5b22d951acb566619372a47932d4c https://get.docker.com out/install2.sh
RUN ls -l /out
EOF

#5 [2/3] ADD --checksum=sha512:c6337ff13c9f58c2016cccf88ace46660b22ade0931b424b4a05d009770782894710e9c61aacd8cd76c53fcb9f5fe42b39c5b22d951acb566619372a47932d4c https://get.docker.com out/install2.sh
#5 ERROR: digest mismatch sha256:f28da2f67cafc5186812966ea4a17ac54b57a9578070aec019e3e297d1aee532: sha512:c6337ff13c9f58c2016cccf88ace46660b22ade0931b424b4a05d009770782894710e9c61aacd8cd76c53fcb9f5fe42b39c5b22d951acb566619372a47932d4c
------
 > [2/3] ADD --checksum=sha512:c6337ff13c9f58c2016cccf88ace46660b22ade0931b424b4a05d009770782894710e9c61aacd8cd76c53fcb9f5fe42b39c5b22d951acb566619372a47932d4c https://get.docker.com out/install2.sh:
------
ERROR: failed to solve: digest mismatch sha256:f28da2f67cafc5186812966ea4a17ac54b57a9578070aec019e3e297d1aee532: sha512:c6337ff13c9f58c2016cccf88ace46660b22ade0931b424b4a05d009770782894710e9c61aacd8cd76c53fcb9f5fe42b39c5b22d951acb566619372a47932d4c

[hoylemd]

Thanks for looking into this folks! It's a bit disappointing to learn that only sha256 is supported after all, but the reasoning makes sense.

[thaJeztah]

@tonistiigi do we have a tracking ticket for the error-handling (see my comment)?