After Update to Docker 25.0.0 (buildx v0.12.1), host-gateway in build does not work anymore #2201

Roemer · 2024-01-23T11:58:40Z

Description

When building an image with the parameter --add-host=host.docker.internal:host-gateway, the build is aborted with the following message:

ERROR: unable to derive the IP value for host-gateway: failed to parse host-gateway IP: <nil>

Reproduce

docker build --add-host=host.docker.internal:host-gateway .

Expected behavior

The docker build works.

docker version

Client: Docker Engine - Community
 Version:           25.0.0
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        e758fe5
 Built:             Thu Jan 18 17:09:59 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          25.0.0
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       615dfdf
  Built:            Thu Jan 18 17:09:59 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.27
  GitCommit:        a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc:
  Version:          1.1.11
  GitCommit:        v1.1.11-0-g4bccb38
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    25.0.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 3
  Running: 2
  Paused: 0
  Stopped: 1
 Images: 8
 Server Version: 25.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc version: v1.1.11-0-g4bccb38
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.15.133.1-microsoft-standard-WSL2
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 19.54GiB
 Name: w00s71
 ID: 82ae4f4e-4d2a-4ce8-94d9-43c43d29095a
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http://localhost:8888
 HTTPS Proxy: http://localhost:8888
 No Proxy: localhost,127.0.0.1,.local
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://redacted/
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

Additional Info

I rolled back to 24.0.7 and everything works again.

VERSION_STRING=5:24.0.7-1~debian.12~bookworm
sudo apt-get install docker-ce=$VERSION_STRING docker-ce-cli=$VERSION_STRING containerd.io docker-buildx-plugin docker-compose-plugin

The text was updated successfully, but these errors were encountered:

thaJeztah · 2024-01-23T12:32:10Z

This error looks to come from buildx (which runs docker build);

buildx/build/utils.go

Lines 75 to 83 in d852568

    
           // If the IP Address is a "host-gateway", replace this value with the 
        
           // IP address provided by the worker's label. 
        
           if ip == mobyHostGatewayName { 
        
           	hgip, err := nodeDriver.HostGatewayIP(ctx) 
        
           	if err != nil { 
        
           		return "", errors.Wrap(err, "unable to derive the IP value for host-gateway") 
        
           	} 
        
           	ip = hgip.String() 
        
           } else {

Possibly related to this PR (which was in the v0.12.1 update of buildx);

build: skip "host-gateway" validation with moby driver #1446

In the meantime, you can try downgrading docker-buildx to the previous version (0.11.2; https://download.docker.com/linux/debian/dists/bookworm/pool/stable/amd64/)

Let me transfer this ticket to the buildx issue tracker.

thaJeztah · 2024-01-23T12:35:27Z

Actually, that PR looks a lot older, so probably not related; might've been a follow-up somewhere 🤔

thaJeztah · 2024-01-23T12:43:47Z

Hm... looking at the code for nodeDriver.HostGatewayIP(ctx),

buildx/driver/docker/driver.go

Lines 112 to 127 in d852568

    
           func (d *Driver) HostGatewayIP(ctx context.Context) (net.IP, error) { 
        
           	d.hostGateway.once.Do(func() { 
        
           		c, err := d.Client(ctx) 
        
           		if err != nil { 
        
           			d.hostGateway.err = err 
        
           			return 
        
           		} 
        
           		defer c.Close() 
        
           		workers, err := c.ListWorkers(ctx) 
        
           		if err != nil { 
        
           			d.hostGateway.err = errors.Wrap(err, "listing workers") 
        
           			return 
        
           		} 
        
           		for _, w := range workers { 
        
           			// should match github.com/docker/docker/builder/builder-next/worker/label.HostGatewayIP const 
        
           			if v, ok := w.Labels["org.mobyproject.buildkit.worker.moby.host-gateway-ip"]; ok && v != "" {

This may be something in the engine (not setting the label by default?);

On Docker Desktop;

docker buildx inspect default
Name:          default
Driver:        docker
Last Activity: 2023-11-21 14:51:38 +0000 UTC

Nodes:
Name:      default
Endpoint:  default
Status:    running
Buildkit:  v0.12.4+3b6880d2a00f
Platforms: linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64
Labels:
 org.mobyproject.buildkit.worker.containerd.namespace: moby
 org.mobyproject.buildkit.worker.containerd.uuid:      1cc22aa9-1955-4d97-96a7-b6153b61614b
 org.mobyproject.buildkit.worker.executor:             containerd
 org.mobyproject.buildkit.worker.hostname:             docker-desktop
 org.mobyproject.buildkit.worker.moby.host-gateway-ip: 192.168.65.254
 org.mobyproject.buildkit.worker.network:              host
 org.mobyproject.buildkit.worker.selinux.enabled:      false
 org.mobyproject.buildkit.worker.snapshotter:          overlayfs

On a debian machine;

docker buildx inspect default
Name:          default
Driver:        docker
Last Activity: 2024-01-22 11:09:41 +0000 UTC

Nodes:
Name:      default
Endpoint:  default
Status:    running
Buildkit:  v0.12.4+3b6880d2a00f
Platforms: linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
Labels:
 org.mobyproject.buildkit.worker.moby.host-gateway-ip: <nil>

Roemer · 2024-01-23T12:45:27Z

With docker 24.0.7 I get the correct label:

docker buildx inspect default
Name:          default
Driver:        docker
Last Activity: 2024-01-23 11:59:40 +0000 UTC

Nodes:
Name:      default
Endpoint:  default
Status:    running
Buildkit:  v0.11.7+d3e6c1360f6e
Platforms: linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
Labels:
 org.mobyproject.buildkit.worker.moby.host-gateway-ip: 172.17.0.1

thaJeztah · 2024-01-23T12:48:56Z

Label is set here; and it looks like it gets the information from opt.DNSConfig.HostGatewayIP (so perhaps it's only getting a value if it's explicitly set in daemon.json, and doesn't fallback to "gateway" address); https://github.com/moby/moby/blob/178651733852a26bc11e6fa272b48ff37ecc7447/builder/builder-next/controller.go#L446-L452

func getLabels(opt Opt, labels map[string]string) map[string]string {
	if labels == nil {
		labels = make(map[string]string)
	}
	labels[wlabel.HostGatewayIP] = opt.DNSConfig.HostGatewayIP.String()
	return labels
}

Edit: that code was added in moby/moby@21e50b8

builder: pass host-gateway IP as worker label moby/moby#45767

Either it needs a fallback, or it's depending on the gateway-address to be set, which may not yet be the case at that point (done through setHostGatewayIP: https://github.com/moby/moby/blob/178651733852a26bc11e6fa272b48ff37ecc7447/daemon/daemon_unix.go#L897-L910

// setHostGatewayIP sets cfg.HostGatewayIP to the default bridge's IP if it is empty.
func setHostGatewayIP(controller *libnetwork.Controller, config *config.Config) {
        if config.HostGatewayIP != nil {
                return
        }
        if n, err := controller.NetworkByName(network.NetworkBridge); err == nil {
                v4Info, v6Info := n.IpamInfo()
                if len(v4Info) > 0 {
                        config.HostGatewayIP = v4Info[0].Gateway.IP
                } else if len(v6Info) > 0 {
                        config.HostGatewayIP = v6Info[0].Gateway.IP
                }
        }
}

The gateway IP is set when initializing the network controller, which happens somehwere in NewDaemon -> daemon.restore() https://github.com/moby/moby/blob/178651733852a26bc11e6fa272b48ff37ecc7447/cmd/dockerd/daemon.go#L261
BuildKit worker is initialized in newRouterOptions in DaemonCLI.start() https://github.com/moby/moby/blob/178651733852a26bc11e6fa272b48ff37ecc7447/cmd/dockerd/daemon.go#L304-L307

Roemer · 2024-01-23T12:51:28Z

There are other issues around DNS with Docker 25, like docker/cli#4816 (comment)
Maybe that is relevant?

thaJeztah · 2024-01-23T12:57:07Z

As a workaround you can manually configure the IP-address to use; https://docs.docker.com/engine/reference/commandline/dockerd/#configure-host-gateway-ip. I see the docs only mentions the --host-gateway-ip flag, but it can be set in the daemon.json (we should fix the docs for that).

For example;

cat /etc/docker/daemon.json

{
  "host-gateway-ip": "172.17.0.1"
}

After that, make sure to restart the daemon to reload the config (also something we should look if we can do so on "reload" (note to self));

systemctl restart docker


docker builder inspect default
Name:          default
Driver:        docker
Last Activity: 2024-01-22 11:09:41 +0000 UTC

Nodes:
Name:      default
Endpoint:  default
Status:    running
Buildkit:  v0.12.4+3b6880d2a00f
Platforms: linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
Labels:
 org.mobyproject.buildkit.worker.moby.host-gateway-ip: 172.17.0.1

thaJeztah · 2024-01-23T12:58:22Z

cc @dvdksn for the documentation 😅

Roemer · 2024-01-23T13:00:33Z

I'm fine staying with 24.x for now, just wanted to let you know that this did break with 25.0.0.

dvdksn · 2024-01-23T13:03:14Z

@thaJeztah the docs above do mention host-gateway-ip, see 2nd paragraph in the linked section. I could swap out the example to show the config file instead of the flag, I guess

thaJeztah · 2024-01-24T10:58:43Z

@Roemer v25.0.1 was released, and should fix your issue (see #2204)

Roemer · 2024-01-24T11:02:15Z

Thanks for the quick reaction! I can confirm that this is now correctly working with 25.0.1.

elichai · 2024-12-22T19:01:28Z

It still doesn't work with the docker-container driver: https://github.com/elichai/docker-bug/actions/runs/12456873279/job/34770817326

Roemer added the kind/bug Something isn't working label Jan 23, 2024

thaJeztah transferred this issue from docker/cli Jan 23, 2024

thaJeztah changed the title ~~After Update to Docker 25.0.0, host-gateway in build does not work anymore~~ After Update to Docker 25.0.0 (buildx v0.12.1), host-gateway in build does not work anymore Jan 23, 2024

dvdksn mentioned this issue Jan 23, 2024

docs: update host-gateway-ip to use daemon.json instead of cli flag docker/cli#4818

Merged

This was referenced Jan 23, 2024

fix "host-gateway-ip" label not set for builder workers moby/moby#47187

Merged

[25.0 backport] fix "host-gateway-ip" label not set for builder workers moby/moby#47192

Merged

dvdksn mentioned this issue Jan 23, 2024

[25.0 backport] docs: update host-gateway-ip to use daemon.json instead of cli flag docker/cli#4819

Merged

thaJeztah closed this as completed in moby/moby#47187 Jan 23, 2024

thaJeztah mentioned this issue Jan 24, 2024

Dockerfile: update to Docker Engine v25.0.1 #2204

Merged

This was referenced Feb 5, 2024

'extra_hosts' not working anymore in 'docker stack deploy' docker/compose#11463

Closed

'extra_hosts' not working anymore in 'docker stack deploy' docker/cli#4859

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

After Update to Docker 25.0.0 (buildx v0.12.1), host-gateway in build does not work anymore #2201

After Update to Docker 25.0.0 (buildx v0.12.1), host-gateway in build does not work anymore #2201

Roemer commented Jan 23, 2024

thaJeztah commented Jan 23, 2024

thaJeztah commented Jan 23, 2024

thaJeztah commented Jan 23, 2024

Roemer commented Jan 23, 2024

thaJeztah commented Jan 23, 2024 •

edited

Loading

Roemer commented Jan 23, 2024

thaJeztah commented Jan 23, 2024 •

edited

Loading

thaJeztah commented Jan 23, 2024

Roemer commented Jan 23, 2024

dvdksn commented Jan 23, 2024

thaJeztah commented Jan 24, 2024

Roemer commented Jan 24, 2024

elichai commented Dec 22, 2024

After Update to Docker 25.0.0 (buildx v0.12.1), host-gateway in build does not work anymore #2201

After Update to Docker 25.0.0 (buildx v0.12.1), host-gateway in build does not work anymore #2201

Comments

Roemer commented Jan 23, 2024

Description

Reproduce

Expected behavior

docker version

docker info

Additional Info

thaJeztah commented Jan 23, 2024

thaJeztah commented Jan 23, 2024

thaJeztah commented Jan 23, 2024

Roemer commented Jan 23, 2024

thaJeztah commented Jan 23, 2024 • edited Loading

Roemer commented Jan 23, 2024

thaJeztah commented Jan 23, 2024 • edited Loading

thaJeztah commented Jan 23, 2024

Roemer commented Jan 23, 2024

dvdksn commented Jan 23, 2024

thaJeztah commented Jan 24, 2024

Roemer commented Jan 24, 2024

elichai commented Dec 22, 2024

thaJeztah commented Jan 23, 2024 •

edited

Loading

thaJeztah commented Jan 23, 2024 •

edited

Loading