Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

grpc: failed to unmarshal the received message: string field contains invalid UTF-8 #2850

Open
xmapst opened this issue Dec 10, 2024 · 8 comments
Open

grpc: failed to unmarshal the received message: string field contains invalid UTF-8 #2850

xmapst opened this issue Dec 10, 2024 · 8 comments
Labels
status/needs-more-info

Comments

@xmapst
Copy link

xmapst commented Dec 10, 2024 •
edited
Loading

After upgrading to buildx version v0.19.2, the following error is reported when building the image

[+] Building 3.2s (5/20)                                                                                                                                                                                                  docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                                                0.0s
 => => transferring dockerfile: 829B                                                                                                                                                                                                0.0s
 => [internal] load metadata for nexus.internal.com/base-image:linux-v2.1.18                                                                                                                                               0.1s
 => [internal] load .dockerignore                                                                                                                                                                                                   0.0s
 => => transferring context: 2B                                                                                                                                                                                                     0.0s
 => CANCELED [ 1/16] FROM nexus.internal.com/base-image:linux-v2.1.18@sha256:084539c767645814983a002ca411bfd2a0d80db871d19459f171481fdacf5e3b                                                                              2.8s
 => => resolve nexus.internal.com/base-image:linux-v2.1.18@sha256:084539c767645814983a002ca411bfd2a0d80db871d19459f171481fdacf5e3b                                                                                         0.1s
 => => sha256:084539c767645814983a002ca411bfd2a0d80db871d19459f171481fdacf5e3b 2.85kB / 2.85kB                                                                                                                                      0.0s
 => => sha256:2d473b07cdd5f0912cd6f1a703352c82b512407db6b05b43f2553732b55df3bc 76.10MB / 76.10MB                                                                                                                                    1.9s
 => => sha256:777c67696248a1e3d74fb3a1be418d3a393beb8bb362370faa659a282a336006 5.35MB / 5.35MB                                                                                                                                      0.4s
 => => sha256:d6bfbb7dc215c3a340544c4c87042210e647b140d51a6f210541801fce8274fd 7.78kB / 7.78kB                                                                                                                                      0.0s
 => => sha256:1ff5115dcffa2e271b232c9dde16c0f7f399773da441c440b497381167582a5f 59.59MB / 142.52MB                                                                                                                                   2.9s
 => => sha256:11006e65265e13aa0d3f6d56ce11dee5dca992cdcb8a9760431a109aaa79e375 15.73MB / 38.09MB                                                                                                                                    2.9s
 => => sha256:374999033e8d93a2d24cb2b13d62c8240959ab05c848b8fef1de101a600e7adb 20.97MB / 98.93MB                                                                                                                                    2.9s
 => => sha256:977e16a77d624fa14338325b88ad9f2aa3059564fa2daac0dce60f8dee15f0dc 2.64MB / 2.64MB                                                                                                                                      1.1s
 => => sha256:b257af7807da588a476e41081426372189b78526e7c371de2680606c3c02822f 315.64kB / 315.64kB                                                                                                                                  0.5s
 => => sha256:7e226f4e7f678e9028d1b76c8c0d56d6900058be170ce69baa63ba5567f8f53a 14.68MB / 901.58MB                                                                                                                                   2.9s
 => => sha256:08be1ee72a529c98177917e0c10bbbc2713519220fad9b3f810b691249a2faa9 193B / 193B                                                                                                                                          0.5s
 => => sha256:91e577162b472fd8eacc65cf3641be7d736a6d285185d878119e7ae53e4079cd 124B / 124B                                                                                                                                          0.5s
 => => sha256:8cc69cd3c0537c8a40164bff8c41e624703873f6e31d529d6316ada963d6fa2b 60.82MB / 172.92MB                                                                                                                                   2.9s
 => => sha256:a941cf7ebf1a67f1c76848a42b49d08eaec0fc9b3d62e4133f2e9d547f1d44f9 15.37MB / 15.37MB                                                                                                                                    2.4s
 => => extracting sha256:2d473b07cdd5f0912cd6f1a703352c82b512407db6b05b43f2553732b55df3bc                                                                                                                                           0.9s
 => ERROR [internal] load build context                                                                                                                                                                                             2.5s
 => => transferring context: 121.49MB                                                                                                                                                                                               2.4s
------
 > [internal] load build context:
------
ERROR: failed to solve: Internal: rpc error: code = Internal desc = grpc: failed to unmarshal the received message: string field contains invalid UTF-8
3057857  /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
github.com/moby/buildkit/session/filesync.recvDiffCopy
        /root/build-deb/engine/vendor/github.com/moby/buildkit/session/filesync/diffcopy.go:104
github.com/moby/buildkit/session/filesync.FSSync
        /root/build-deb/engine/vendor/github.com/moby/buildkit/session/filesync/filesync.go:236
github.com/moby/buildkit/source/local.(*localSourceHandler).snapshot
        /root/build-deb/engine/vendor/github.com/moby/buildkit/source/local/source.go:264
github.com/moby/buildkit/source/local.(*localSourceHandler).Snapshot
        /root/build-deb/engine/vendor/github.com/moby/buildkit/source/local/source.go:153
github.com/moby/buildkit/solver/llbsolver/ops.(*SourceOp).Exec
        /root/build-deb/engine/vendor/github.com/moby/buildkit/solver/llbsolver/ops/source.go:108
github.com/moby/buildkit/solver.(*sharedOp).Exec.func2
        /root/build-deb/engine/vendor/github.com/moby/buildkit/solver/jobs.go:1100
github.com/moby/buildkit/util/flightcontrol.(*call[...]).run
        /root/build-deb/engine/vendor/github.com/moby/buildkit/util/flightcontrol/flightcontrol.go:122
sync.(*Once).doSlow
        /usr/local/go/src/sync/once.go:74
sync.(*Once).Do
        /usr/local/go/src/sync/once.go:65
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1695

3066149 v0.19.2 /usr/libexec/docker/cli-plugins/docker-buildx buildx build --debug --network host -t test:test -f ../Dockerfile .
google.golang.org/grpc.(*ClientConn).Invoke
        google.golang.org/grpc@v1.66.3/call.go:35
github.com/moby/buildkit/api/services/control.(*controlClient).Solve
        github.com/moby/buildkit@v0.18.0/api/services/control/control_grpc.pb.go:88
github.com/moby/buildkit/client.(*Client).solve.func2
        github.com/moby/buildkit@v0.18.0/client/solve.go:269
golang.org/x/sync/errgroup.(*Group).Go.func1
        golang.org/x/sync@v0.8.0/errgroup/errgroup.go:78
runtime.goexit
        runtime/asm_amd64.s:1695

3057857  /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
github.com/moby/buildkit/solver.(*edge).execOp
        /root/build-deb/engine/vendor/github.com/moby/buildkit/solver/edge.go:966
github.com/moby/buildkit/solver/internal/pipe.NewWithFunction[...].func2
        /root/build-deb/engine/vendor/github.com/moby/buildkit/solver/internal/pipe/pipe.go:78
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1695

3066149 v0.19.2 /usr/libexec/docker/cli-plugins/docker-buildx buildx build --debug --network host -t test:test -f ../Dockerfile .
github.com/moby/buildkit/client.(*Client).solve.func2
        github.com/moby/buildkit@v0.18.0/client/solve.go:285
golang.org/x/sync/errgroup.(*Group).Go.func1
        golang.org/x/sync@v0.8.0/errgroup/errgroup.go:78

version info:

# docker buildx version
github.com/docker/buildx v0.19.2 1fc5647
# docker buildx ls
NAME/NODE     DRIVER/ENDPOINT   STATUS    BUILDKIT   PLATFORMS
default*      docker                                 
 \_ default    \_ default       running   v0.17.3    linux/amd64 (+3), linux/386
# docker info
Client: Docker Engine - Community
 Version:    27.4.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.19.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.31.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 6
  Running: 6
  Paused: 0
  Stopped: 0
 Images: 6
 Server Version: 27.4.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 88bf19b2105c8b17560993bee28a01ddc2f97182
 runc version: v1.2.2-0-g7cb3632
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.10.0-26-amd64
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.88GiB
........
@tonistiigi
Copy link
Member

After upgrading to buildx version v0.19.2,

Upgrading from what version?

Please post reproduction steps.

@nul0op
Copy link

nul0op commented Dec 23, 2024 •
edited
Loading

Hello, we get the same problem since an upgrade this WE. 27.3.1 -> 27.4.1
end result for a build that worked fine before:

#0 building with "default" instance using docker driver
[...]
#6 [application  3/15] RUN [...redacted...]
#6 CACHED
#7 [application internal] load build context
#7 transferring context: 848.24kB done
#7 ERROR: rpc error: code = Internal desc = grpc: failed to unmarshal the received message: string field contains invalid UTF-8
#8 [application  4/15] RUN apt-get update
#8 0.334 runc run failed: container process is already dead
#8 CANCELED
------
 > [application internal] load build context:
------
failed to solve: Internal: rpc error: code = Internal desc = grpc: failed to unmarshal the received message: string field contains invalid UTF-8

version that works with the same build:

Client: Docker Engine - Community
 Version:           27.3.1
 API version:       1.47
 Go version:        go1.22.7
 Git commit:        ce12230
 Built:             Fri Sep 20 11:41:11 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.3.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.7
  Git commit:       41ca978
  Built:            Fri Sep 20 11:41:11 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.22
  GitCommit:        7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc:
  Version:          1.1.14
  GitCommit:        v1.1.14-0-g2c9f560
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.17.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx

version that FAILS with the same build:

Client: Docker Engine - Community
 Version:           27.4.1
 API version:       1.47
 Go version:        go1.22.10
 Git commit:        b9d17ea
 Built:             Tue Dec 17 15:45:56 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.4.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.10
  Git commit:       c710b88
  Built:            Tue Dec 17 15:45:56 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.24
  GitCommit:        88bf19b2105c8b17560993bee28a01ddc2f97182
 runc:
  Version:          1.2.2
  GitCommit:        v1.2.2-0-g7cb3632
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.19.3
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx

if we can help pinpoint the problem, please ask

@nul0op
Copy link

nul0op commented Dec 23, 2024 •
edited
Loading

some tests i did:

  • changed to base image in the pipeline => same error
  • changed the layer just before the problem (by saving a file somewhere), just to try to avoid a layer cache issue
  • on the same OS release, same dockerfile, i didn't have the issue. only difference there: i didn't build the same Dockerfile before the os upgrade (i mean with a lower docker version)
  • split "docker compose --build" in a separate docker build call, with the --no-cache flag => same issue

so for me, it's only happening while docker compose build is running under the gitlab-ce runner hood, on a debian bookworm host with latest docker-ce packages (as of today)

again, if i can test something else, just ask

@xmapst
Copy link
Author

xmapst commented Jan 3, 2025

@tonistiigi Any progress?

@tonistiigi
Copy link
Member

My best guess is that you have something in the build context that makes the paths or symlink fields in https://github.com/tonistiigi/fsutil/blob/master/types/stat.proto#L10-L17 contain values that are not strings. Not sure what case would do that but likely that our validation that these are actual strings and not non-printable bytes is more strict now.

@xmapst
Copy link
Author

xmapst commented Jan 3, 2025

My best guess is that you have something in the build context that makes the paths or symlink fields in https://github.com/tonistiigi/fsutil/blob/master/types/stat.proto#L10-L17 contain values that are not strings. Not sure what case would do that but likely that our validation that these are actual strings and not non-printable bytes is more strict now.

After checking, it was found that some file names were indeed not encoded in UTF-8, but in GB2312.

@nul0op
Copy link

nul0op commented Jan 13, 2025

i confirm it's related to a filename encoding.
As a workaround we have excluded some folders from the image (.dockerignore) and it builds fine now.

@tonistiigi
Copy link
Member

Do you have a practical testcase for this, so we can look if there is a way to mitigate with workaround or improve error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
status/needs-more-info
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@tonistiigi @crazy-max @nul0op @xmapst