network mode "custom_network" not supported by buildkit #175
Comments
|
Sorry, I'm not sure if we will ever start supporting this as it makes the build dependant on the configuration of a specific node and limits the build to a single node. |
That horse has bolted - SSH mount makes the build dependent upon the configuration of a single node - where did that dogma even get started? |
No, it does not. You can forward your ssh agent against any node or a cluster of nodes in buildx. Not really different than just using private images. |
Why would someone do that? ssh-agent is a something that needs to be fairly well locked down - why would someone forward it across an insecure connection? I mean, that's a tangent anyway. Being able to run integration-tests in a docker build was an incredibly useful feature, one less VM to spin up, and one less iceberg to melt, it's just useful because it's efficient. It's also great to not have to run nodejs, ruby, etc on the build host but instead just have them as container dependency, if you can do all your tests in a docker build container it's one less thing to lock down. Anyhow, I apologise for running off on a tangent. All I'm saying is, it would be awesome if you could bring that functionality into the latest version of docker along with the means to temporary mount secrets. It's just a really lightweight way to run disposable VMs without touching the host or even giving any rights to run any scripts or anything on the host. |
Why would that connection be insecure? Forwarding agent is more secure than build secrets because your nodes never get access to your keys.
We have solutions for build secrets, privileged execution modes (where you needed |
But I'd like to spin up a network for each build - and have all the stuff running that would be needed for the integration tests. But again, I have to loop back around and either do weird stuff with iptables, or run postgres on the host and share it with all builds (contention/secrets/writing to the same resources/etc). You could see how it would be so much more encapsulated and attractive if I could spin up a network per build with a bunch of stub services and tear it down afterwards ? |
I'm talking about the socat hack where you forward the socket over TCP - you might have been referring to something else. |
|
moby/buildkit#1337 sounds cool but honestly, given the choice between something that right now works or something that will drop in 2 years time, I know what most of the community would choose. |
https://medium.com/@tonistiigi/build-secrets-and-ssh-forwarding-in-docker-18-09-ae8161d066 |
Nah your secrets and forwarding feature is great - love it. Rocker had secrets support 3 years ago but that project withered on the vine. |
|
The sidecar also sounds great and very clever and well structured. But again, 3 years ago I could build with secrets and talk to network services to run integration tests. |
Also, it does work in compose while build secrets does not. |
|
Adding another use case where specifying the network would be useful: "hermetic builds". I'm defining a docker network with I can do this with the classic docker build today, or I can create an entire VM with the appropriate network settings, perhaps it would also work if I setup a DinD instance, but it would be useful for buildkit to support this natively. |
Good point, I should have mentioned I was doing that too for git dependencies, and... Docker themselves have blogged about using it to augment the docker cache. Now I just burn the network, take lots of coffee breaks, and do my bit to melt the ice caps. |
|
@bryanhuntesl The proxy vars are still supported. For this use case, cache mounts might be a better solution now https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/syntax.md#run---mounttypecache |
|
This is particularly needed in environments such as Google Cloud Build where ambient credentials (via special-IP metadata service) are available only on a particular named network, not on the default network, in order to keep their exposure to build steps opt-in. |
|
Any updates on this? I have also checked this issue moby/buildkit#978, but can't find a straight answer. I've disabled |
|
The recommendation is to use |
|
Thank you! It seemed like this was a weird use case, but it fits my needs for now. I'll be looking for a better solution, but in the meanwhile I'll use the recommendation. |
Anyone have a working example of this in Github Actions? Not working for me. Run docker/setup-buildx-action@v1
with:
install: true
buildkitd-flags: --debug
driver-opts: network=custom-network
driver: docker-container
use: true
env:
DOCKER_CLI_EXPERIMENTAL: enabled
Docker info
Creating a new builder instance
/usr/bin/docker buildx create --name builder-3eaacab9-d53e-490c-9020-xxx --driver docker-container --driver-opt network=custom-network --buildkitd-flags --debug --use
builder-3eaacab9-d53e-490c-9020-bae1d022b444
Booting builder
Setting buildx as default builder
Inspect builder
BuildKit version
moby/buildkit:buildx-stable-1 => buildkitd github.com/moby/buildkit v0.9.3 8d2625494a6a3d413e3d875a2ff7xxxBuild
/usr/bin/docker build -f Dockerfile -t my_app:latest --network custom-network --target production .
time="2022-01-19T17:00:XYZ" level=warning msg="No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load"
error: network mode "custom-network" not supported by buildkit. You can define a custom network for your builder using the network driver-opt in buildx create.
Error: The process '/usr/bin/docker' failed with exit code 1 |
I don't see how that setup is any different that my configuration. Am I missing something?
/usr/bin/docker buildx create --name builder-3eaacab9-d53e-490c-9020-xxx --driver docker-container --driver-opt network=custom-network --buildkitd-flags --debug --useHere's the network create: /usr/bin/docker network create custom-network
35bb341a1786f50af6b7baf7853ffc46926b62739736e93709e320xxx
/usr/bin/docker run --name my_container --network custom-network |
You don't pass the custom network name with build commands. Your builder instance is already part of that network. |
|
OK, so once you've got it set up, how do you get name resolution to work? If I have a container |
|
I have the same problem as @philomory. Name resolution doesn't work. Builder has been created with the following command:
|
|
It seems GCE's metadata server IP is docker buildx create --name builder --driver docker-container --driver-opt network=cloudbuild --use
docker buildx build \
--add-host metadata.google.internal:169.254.169.254 \
... \
.and inside RUN curl "http://metadata.google.internal/computeMetadata/v1/project/project-id" -H "Metadata-Flavor: Google" |
|
Thanks for the tips @fibbers, it works like a charm. It will do the job until a real fix. |
|
@tonistiigi What's the right way to use the
I'm currently doing something like Using |
|
I've been seeing similar. You can run the build in a user specified network. But the buildkit container on that network has DNS set to the docker's localhost entry which won't get passed through to nested containers. So the RUN steps within the build don't have that DNS resolution. I'm not sure of the best way to get that to pass through, perhaps a proxy running in the buildkit container that lets DNS get set to the container IP instead of localhost? |
Please keep such comments at the door. No need for this. I'll skip over some of you other wordings, because I don't think there's anything constructive in those
There's many things associated with this, and this feature is not trivial to implement (if possible at all). Let me try to do a short summary from the top of my head. Note that I'm not a BuildKit maintainer; I'm familiar with the overal aspects, but not deeply familiar with all parts of BuildKit. First of all, buildx (the repository this ticket is posted in) is a client for BuildKit; the feature requested here would have to be implemented by the builder (BuildKit); Ideally this ticket would be moved to a more appropriate issue tracker, but unfortunately GitHub does not allow transferring tickets between orgs; in addition, multiple projects may need to be involved. There's no harm in having a ticket in this repository (nor to have discussions elsewhere), but ultimately this feature would have to be both accepted by upstream BuildKit and/or Moby maintainers, and implemented respectivee projects. Now the more technical aspect of this feature request (or: this is where the fun starts). BuildKit can be run in different settings / ways;
Depending on the above, semantics, and available features will differ. Standalone BuildKit buildersStandalone builders are designed to be stateless; builders may have cache from prior builds, but don't persist images (only build-cache), nor do they have a concept of "networks" (other than "host" networking or "no" networking). The general concept here is that standalone builders can be ephemeral (auto-scaling cluster), and that builds don't depend on prior state. They may advertise certain capabilities (BuildKit API version, native platform/architecture), but otherwise builders are interchangeable (perform a build, and export the result (e.g. push to a registry, or export as an OCI bundle). Given that standalone builders don't have access to "docker custom networks" (there's no docker daemon involved), it won't be possible to provide a per build option to use a specific network. The workaround mentioned in this thread is to;
In this configuration, the "host" (container) can resolve containers running in that custom network, but this only works in very specific scenarios;
All of the above combined make this a workaround that would work in only very specific scenarios. Implementing this as a native feature for standalone builders would quickly go down the rabbit-hole; "custom network attached to this builder" as well as "state of dependencies" would have to be exposed as capabilities, so that buildx can query which builder to select for a specific build, or an external orchestrator would need to be involved to construct builders (and dependencies) on-demand. Both would be moving away significantly from the current design of standalone builders (stateless, ephemeral). As part of the Moby daemon ("Docker Engine")When using the "default' builder on Docker Engine, BuildKit is not running as a standalone daemon, but compiled into the Docker Engine. In this scenario, BuildKit has some (but limited) access to features provided by the Docker Engine. For example, it's possible to use images that are available in the local image cache as part of the build (e.g. an image you built earlier, but that's not pushed to a registry). There's also limitations; when using "graphdrivers", the Docker Engine does not provide a multi-arch image store, so it's not possible to build multple architectures in a single build. (This will be possible in the near future and being worked on as part of the containerd image store integration). Containers created during build are optimized for performance; containers used for build-steps tend to be very short-lived. "regular" containers as created by the Docker Engine have a non-insignificant overhead to provide all features that Long story short; build-time-containers are (by design) isolated from "regular" containers; they're not managed by the docker daemon itself, won't show up in So, while the "embedded" BuildKit may have more potential options to integrate with custom networks, integrating with custom networks will require a significant amount of work; both in BuildKit (integration with the network stack) and in Moby / "docker engine" to (somehow) allow BuildKit to create different "flavors" of containers (isolated / non-isolated). This will come with a performance penalty, in addition to complexity involved (a secondary set of (short-lived) containers that can be attached to a network, but are not directly managed by the Moby / Docker Engine itself. |
|
Thank you for the analysis. |
|
TLDR; Which scenario would be better? EDIT: For me personally - build time is not the primary thing I am after. I wish the solution to work. Currently using the classic builder still works, but my primary concern is that this feature disappears. |
|
Thank you @thaJeztah for that clarifying summary. I wonder if it'd be worth a feature request against Docker Compose, which seems like it can do most of what the "containerised builder workaround" needs, since people in this thread have tried to do it that way already. If Docker Compose was able to create/use/remove temporary private builder instances on its own (or the config-specified) network, along with resolving the apparent issue that a standalone BuildKit instance in a Docker container attached to a custom network doesn't get the right DNS setup for services on that custom network (see #175 (comment) from earlier attempts to use the "workaround" manually), then I think it can deliver the "workaround" flow fairly naturally for some of the use-cases described here. I see a similar idea mentioned at docker/compose#10745 (comment) but I think that was about selecting an existing builder, rather than instantiating a new one. compose-spec/compose-spec#386 is along these lines but it wants to be super-generic; it is probably too generic for this use-case, particularly if we want to tear-down the builder instance after usage or at compose-down time. In this case, the builder is more like another service in Docker Compose that compose knows how to use when building service images. (That also might be a better way to visualise and implement it, similar to existing That is separate from the existing That said, I'm not a Docker Compose user, so I may be overestimating what it can cover, or misunderstanding the relevant workflow. Either way, unless this is an obviously-faulty idea to Docker Compose users, it'd be better to discuss in a ticket there, to focus on the relevant use-cases that involve Docker Compose, and also focus this ticket on buildx-involved use-cases. |
There's no immediate plans to actively remove the classic builder, but no active development is happening on it; consider it in "maintenance mode", and mostly to support building native Windows containers (BuildKit does not yet support Windows Containers, although work on that is in progress). The classic Builder's architecture does not give a lot of room for further expansion, so it will start to diverge / get behind BuildKit more over time. The classic builder may also not make the transition to the containerd image-store integration; there's currently a very rudimentary implementation to help the transition, but we're aware of various limitations in that implementations that may not be addressable with the classic buidler.
Perhaps it'd be useful to start a GitHub discussion; GitHub tickets aren't "great" for longer conversations and don't provide threads (not sure which repository would be best; perhaps BuildKit (https://github.com/moby/buildkit/discussions) to collect slightly more in-depth information about use-cases. I know there was a lot of contention around the original implementation, which also brought up concerns about portability and the feature being out of scope for building (see lengthy discussion on moby/moby#10324 and moby/moby#20987 (carried in moby/moby#27702)). Use-cases that I'm aware of;
But there may be other use-cases. Some of those may make more sense in a "controlled" environment (local / special purpose machine; single user), but get complicated fast in other environments. Having more data about use-cases could potentially help design around those (which may be through a different approach). |
|
To repeat a nearly 2 year old comment here, buildkit does support running with a custom network, it's even documented: https://docs.docker.com/build/drivers/docker-container/#custom-network The issue isn't that it won't run on a custom network, instead, as so often happens on the internet, it was DNS. When you run a container runtime (which buildkit does) inside of a container on a custom network, it sees the DNS settings of that parent container: $ docker network create build
9b7c83ceda7e2552e99d27c29d275936e882fd9cc9488361209bbf4421c2f180
$ docker run -it --rm --net build busybox cat /etc/resolv.conf
search lan
nameserver 127.0.0.11
options ndots:0And as docker and other runtimes do, they refuse to use 127.0.0.11 as a DNS server, so the nested container falls back to 8.8.8.8. Here's a demo for proof: #!/bin/sh
set -ex
docker network create custom-network
echo "hello from custom network" >test.txt
docker run --name "test-server" --net custom-network -d --rm \
-v "$(pwd)/test.txt:/usr/share/nginx/html/test.txt:ro" nginx
server_ip="$(docker container inspect test-server --format "{{ (index .NetworkSettings.Networks \"custom-network\").IPAddress }}")"
echo "Server IP is ${server_ip}"
cat >Dockerfile <<EOF
FROM curlimages/curl as build
USER root
ARG server_ip
RUN mkdir -p /output \
&& cp /etc/resolv.conf /output/resolv.conf \
&& echo \${server_ip} >/output/server_ip.txt \
&& (curl -sSL http://test-server/test.txt >/output/by-dns.txt 2>&1 || :) \
&& (curl -sSL http://\${server_ip}/test.txt >/output/by-ip.txt 2>&1 || :)
FROM scratch
COPY --from=build /output /
EOF
docker buildx create \
--name custom-net-build \
--driver docker-container \
--driver-opt "network=custom-network"
docker buildx build --builder custom-net-build --build-arg "server_ip=${server_ip}" \
-o "type=local,dest=output" .
docker buildx rm custom-net-build
docker stop test-server
docker network rm custom-networkRunning that shows that custom networks are supported, just not DNS: $ ./demo-custom-network.sh
+ docker network create custom-network
bd5ce7361f5fc94b0da0fe32a3f5482176a6fcaca68997556d3449269c451cea
+ echo hello from custom network
+ pwd
+ docker run --name test-server --net custom-network -d --rm -v /home/bmitch/data/docker/buildkit-network/test.txt:/usr/share/nginx/html/test.txt:ro nginx
31bf9516fdc6dba7d97796be6b6c55f2a134a3050a7b1286a2ac96658e444c62
+ docker container inspect test-server --format {{ (index .NetworkSettings.Networks "custom-network").IPAddress }}
+ server_ip=192.168.74.2
+ echo Server IP is 192.168.74.2
Server IP is 192.168.74.2
+ cat
+ docker buildx create --name custom-net-build --driver docker-container --driver-opt network=custom-network
custom-net-build
+ docker buildx build --builder custom-net-build --build-arg server_ip=192.168.74.2 -o type=local,dest=output .
[+] Building 4.2s (9/9) FINISHED
=> [internal] booting buildkit 1.8s
=> => pulling image moby/buildkit:buildx-stable-1 0.4s
=> => creating container buildx_buildkit_custom-net-build0 1.5s
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 393B 0.0s
=> [internal] load metadata for docker.io/curlimages/curl:latest 0.6s
=> [auth] curlimages/curl:pull token for registry-1.docker.io 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [build 1/2] FROM docker.io/curlimages/curl:latest@sha256:4bfa3e2c0164fb103fb9bfd4dc956facce32 1.2s
=> => resolve docker.io/curlimages/curl:latest@sha256:4bfa3e2c0164fb103fb9bfd4dc956facce32b6c5d4 0.0s
=> => sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3 42B / 42B 0.1s
=> => sha256:fcad2432d35a50de75d71a26d674352950ae2f9de77cb34155bdb570f49b5fc3 4.04MB / 4.04MB 0.8s
=> => sha256:c926b61bad3b94ae7351bafd0c184c159ebf0643b085f7ef1d47ecdc7316833c 3.40MB / 3.40MB 0.8s
=> => extracting sha256:c926b61bad3b94ae7351bafd0c184c159ebf0643b085f7ef1d47ecdc7316833c 0.1s
=> => extracting sha256:fcad2432d35a50de75d71a26d674352950ae2f9de77cb34155bdb570f49b5fc3 0.1s
=> => extracting sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3 0.0s
=> [build 2/2] RUN mkdir -p /output && cp /etc/resolv.conf /output/resolv.conf && echo 192.168 0.2s
=> [stage-1 1/1] COPY --from=build /output / 0.0s
=> exporting to client directory 0.0s
=> => copying files 357B 0.0s
+ docker buildx rm custom-net-build
custom-net-build removed
+ docker stop test-server
test-server
+ docker network rm custom-network
custom-network
$ cat output/resolv.conf
search lan
options ndots:0
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 2001:4860:4860::8888
nameserver 2001:4860:4860::8844
$ cat output/server_ip.txt
192.168.74.2
$ cat output/by-dns.txt
curl: (6) Could not resolve host: test-server
$ cat output/by-ip.txt
hello from custom network |
Hmm.. right, but that should not be the case when using cat /etc/resolv.conf
nameserver 127.0.0.11
options ndots:0
docker run --rm --network=host alpine cat /etc/resolv.conf
nameserver 127.0.0.11
options ndots:0Trying to do the same with a BuildKit container builder that has Creating a custom network and a "test-server" container attached to it; docker network create custom-network
docker run -d --name test-server --network custom-network nginx:alpine
docker container inspect test-server --format '{{ (index .NetworkSettings.Networks "custom-network").IPAddress }}'
172.24.0.2Create a custom builder attached to the network, and allow "host-mode" networking; docker buildx create --name custom-net-build --driver docker-container --driver-opt network=custom-network --buildkitd-flags '--allow-insecure-entitlement network.host'
custom-net-buildRunning a build with docker buildx build --no-cache --builder custom-net-build --network=host --progress=plain --load -<<'EOF'
FROM alpine
RUN cat /etc/resolv.conf
RUN wget http://test-server
EOFHowever, it looks like BuildKit sets the default DNS resolvers, not inheriting from the host; So I think there's something funky going on there, and BuildKit's executor / runtime does not take Had a quick peek at code that I think is related to this; it looks like it logs a message about host networking; https://github.com/moby/buildkit/blob/8849789cf8abdc7d63ace61f8dc548582d22f3b5/executor/runcexecutor/executor.go#L184-L188 But after that unconditionally uses the standard |
|
BuildKit incorrectly replacing localhost DNS resolvers when using host networking is moby/buildkit#3210. There was a PR in progress just over a year ago, but it wasn't completed. moby/buildkit#2404 seems to have had more recent activity, but looks much wider in scope than moby/buildkit#3210. |
Should be solved with moby/buildkit#4524 |
|
If you want to test the fixed BuildKit right now, create a builder with In fact, I made that change in the shell script from #175 (comment) on line 30 --driver-opt "image=moby/buildkit:master,network=custom-network" \and it failed the same way; and I confirmed that it was running BuildKit from commit 2873353. I can see that the BuildKit worker is supposed to be using host-mode networking: So I tried with the extra changes from #175 (comment), resulting in docker buildx create \
--name custom-net-build \
--driver docker-container \
--driver-opt "image=moby/buildkit:master,network=custom-network" \
--buildkitd-flags "--allow-insecure-entitlement network.host"
docker buildx build --builder custom-net-build --build-arg "server_ip=${server_ip}" \
--network host \
-o "type=local,dest=output" .and that worked correctly: $ cat output/resolv.conf
nameserver 127.0.0.11
options ndots:0
$ cat output/by-dns.txt
hello from custom networkSo it'd be nice if the buildx docker-container driver could automatically set up the hosted buildkit with Technically, I guess if BuildKit used the worker's network config rather than the specific I'm not 100% clear on the network layering here. Maybe the Anyway, the relevant buildkit change should be part of the 0.13 release and any pre-releases after 0.13b3, and automatically picked up by |
|
Okay, results of discussion with BuildKit maintainer on that PR is that making this smoother is a BuildX thing, as BuildKit is changing its default network config such that For reference, here is my current working mod of #175 (comment) #!/bin/sh
set -ex
docker network create custom-network
echo "hello from custom network" >test.txt
docker run --name "test-server" --net custom-network -d --rm \
-v "$(pwd)/test.txt:/usr/share/nginx/html/test.txt:ro" nginx
server_ip="$(docker container inspect test-server --format "{{ (index .NetworkSettings.Networks \"custom-network\").IPAddress }}")"
echo "Server IP is ${server_ip}"
cat >Dockerfile <<EOF
FROM curlimages/curl as build
USER root
ARG server_ip
ADD http://${server_ip}/test.txt /output/by-ip-add.txt
ADD http://test-server/test.txt /output/by-dns-add.txt
RUN mkdir -p /output \
&& cp /etc/resolv.conf /output/resolv.conf \
&& echo \${server_ip} >/output/server_ip.txt \
&& (curl -sSL http://test-server/test.txt >/output/by-dns.txt 2>&1 || :) \
&& (curl -sSL http://\${server_ip}/test.txt >/output/by-ip.txt 2>&1 || :)
FROM scratch
COPY --from=build /output /
EOF
docker buildx create \
--name custom-net-build \
--driver docker-container \
--driver-opt "image=moby/buildkit:master,network=custom-network" \
--buildkitd-flags "--allow-insecure-entitlement=network.host --oci-worker-net=host"
docker buildx build --builder custom-net-build --build-arg "server_ip=${server_ip}" \
--network host \
-o "type=local,dest=output" .
docker buildx inspect custom-net-build
docker buildx rm custom-net-build
docker stop test-server
docker network rm custom-network
set +x
for d in output/by-*; do echo -n "$d:"; cat $d; doneThis should remain working even when BuildKit changes default networking to bridge, as I've explicitly passed "--oci-worker-net=host" to the buildkitd in the container. I also added a pair of ADD calls to the Dockerfile, to demonstrate that even if you remove For buildx's docker-container driver, a simple nicety would be for the "custom network" driver option to automatically enable those two buildkitd flags (Thinking about this, the kubernetes driver must be doing something similar, unless the same problem shows up there...) Then the buildx docker-container custom network docs also needs to mention that you need to use Longer term/more-fully, buildx could perhaps use the new bridge mode to expose the custom network to |
|
@TBBle Does that mean that one would be able to use |
|
Edit: Actually no. I tried to get this working, and realised that I don't see how you specify a compose file that actually starts services, and then runs builds that rely on those services; the So I'd need to see a working (and ideally simple) example (i.e. from non-buildx) to understand what I'm trying to recreate. Also, as noted in #175 (comment), compose doesn't currently support creating a buildx builder instance, so the builder instance would need to be created first, which means the network must be created first and then referenced in the compose.yaml as I also just remembered you specified So yeah, without an example of what you think ought to work, I can't really advance any compose/bake discussion further. If you're just using I recrated the same test-script in this style: networks:
custom-network:
# Force the name so that we can reference it when creating the builder
name: custom-network
services:
test-server:
# Force the name so that we can reference from the build-only Dockerfile
container_name: test-server
image: nginx
networks:
- custom-network
volumes:
- type: bind
source: ./test.txt
target: /usr/share/nginx/html/test.txt
read_only: true
build-only:
build:
network: host
dockerfile_inline: |
FROM curlimages/curl as build
USER root
ADD http://test-server/test.txt /output/by-dns-add.txt
RUN mkdir -p /output \
&& cp /etc/resolv.conf /output/resolv.conf \
&& (curl -sSL http://test-server/test.txt >/output/by-dns.txt 2>&1 || :)
RUN for d in /output/by-*; do echo -n "$$d:"; cat $$d; done
FROM scratch
COPY --from=build /output /and then created test.txt with the desired contents ("hello from custom network"), and: $ docker compose up test-server --detach
$ docker buildx create --name custom-net-build --driver docker-container --driver-opt "image=moby/buildkit:master,network=custom-network" --buildkitd-flags "--allow-insecure-entitlement=network.host"
$ docker buildx bake --builder custom-net-build --progress plain --no-cache --set=build-only.output=type=local,dest=output
...
#11 [build 4/4] RUN for d in /output/by-*; do echo -n "$d:"; cat $d; done
#11 0.071 /output/by-dns-add.txt:hello from custom network
#11 0.072 /output/by-dns.txt:hello from custom network
#11 DONE 0.1s
...
$ docker buildx rm custom-net-build
$ docker compose down
$ for d in output/by-*; do echo -n "$d:"; cat $d; done
output/by-dns-add.txt:hello from custom network
output/by-dns.txt:hello from custom networkNote that in the So that seems to work, yeah. If compose and/or bake were able to define builders to create (and ideally agree on the real name of either the builder or the network in order to avoid hard-coding the network name as I did here) then it would just be a relatively simple If that's what you want, then I'd suggest opening a new feature request for it: I think it makes more sense to be part of (If you're feeling really clever, you could include a builder as a normal service in the compose spec, and then use the |
|
@TBBle Thanks; that's also what I found. I appreciate the suggestions but don't yet have the mental energy earmarked in order to complete any. I'll wait a little longer in hopes that someone else streamlines a solution before attempting again :) |
|
My two cents about use cases. I was led here after trying to collapse a complex CI process into a Dockerfile. Specifically, a step runs migrations on a DB, another step dumps the DB's schema to a file, and a third step generates templates using that output. All this work follows the mandate of Docker build recipes—repeatable, portable—but the implementation of spinning up a DB doesn't fit with the paradigm. Reason being: while migrations can run on the DB in step one, the DB itself must be running and accessible for step two, which has to run on a different image. By design, you can't keep a build layer alive once you've moved on to another one. Thus, in my use case, it points to a reasonable need for the DB to be attached as an external service, but still scoped and isolated to the build process. Regrettably, the current tooling ecosystem almost serves this purpose. It provides a Seeking support for the issue, one then reaches this labyrinth of support tickets. And the first thing one sees there, confusingly, is the proverbial “slap of the wrist” for wanting or expecting the feature to exist in the first place. As I see it: Docker builds are supposed to be able to network. We know this. To avoid unpredictable build outputs, the user should use the internet only to download resources that are themselves invariant for a given set of build parameters. If the user breaks this rule, they have earned the consequences. Why should resources isolated to a custom network be any different? Docker networks are an extension of the general concept of networking, and in this context an irreducible one. One of the tolls you pay for moving off of bare metal is that containers need a mediating layer if they want to be able to talk to each other in an un-ambiguous way. It's natural to expect that tool to be available wherever it is needed, and build-time is one of the places where it is sometimes needed. What I’m saying is: this isn’t really about repeatability or portability, is it? It’s about difficulty of implementation. I appreciate that it takes a vast amount of work to satisfy a particular use-case without breaking something else, but that makes it all the more unfortunate to see effort be wasted on framing a difficult use-case as somehow un-“Dockerly”. Especially because it is only difficult as a result of competing priorities taking precedence (BuildKit cache optimizations etc) and especially because of CLI, docs, and GitHub leading developers down a garden path that sometimes ends at that extremely unsatisfying answer. |
|
It's not merely difficulty of implementation; if that were the case, someone would have done it by now. I probably would have done it by now if it was merely "hard to implement". The driving change is that with the introduction of BuildKit and buildx, the structure of the container build system has shifted to be much more strongly isolated, both from outside influences and particularly from Docker. The build system is no longer Docker-specific. This has been part of Docker's direction for years now, as seen by the break-out of moby/moby, OCI standardisation of container formats, etc. The upshot of that for our purposes is that Docker's internal overlay network implementation is not visible to BuildKit ( So wanting to have your builds run against a custom network is not being rejected as "Possible but unDockerly". However, to do that in the current state, you actually need to do it, Docker is no longer in a position to make that work magically for you. So "using buildx containers" is not a work-around, that's the way to achieve what you want. As it's the way to achieve anything you want from buildx that isn't available via the current built-in, limited, (and default) buildkit integration. There's definitely space to improve the UX on "using buildx containers", but it's still going to be the same rough flow of "To do something that we can't do from the built-in buildkit builder, create a custom builder that supports what you need, use it, and tear it down when done" with varying levels of automation around it. Docker and |
|
@MattiasMartens Very well said. I would just add that it was so easy and simple previously with the classic builder. You don't know what you'd be missing until it is deprecated! I must say I don't follow @TBBle's thought process as well as Mattias's. First, because I don't consider it "magic" to make a network available to a container. And second because "the structure of the container build system has shifted to be much more strongly isolated" - meaning isolated from what exactly? Clearly not the internet. I'm not claiming I'm owed a more streamlined compose experience, but I do feel it is important to correctly communicate the experience - as Mattias has. This design short circuits the UX of docker compose for users that run containers that need access to other containers or networks at build time. What happens, for example, when docker detects that a newer version of the container should be running from |
|
@TBBle If I understand correctly, you’re saying it’s not merely “hard to implement” but impossible to implement because of a decision about separation of concerns made by BuildKit? I think I follow your explanation broadly but there are some unfortunate choices of phrase... For example "Docker is no longer in a position to make [custom networks in the build environment] work magically for you" is a strange way to phrase "Docker no longer allows one feature, provided by Docker, through the CLI, to leverage another feature, provided by Docker, through the CLI". If that’s "magic" then we are all wizards! I would be much happier with the status quo if someone, in the five years of this issue being open, had reported success in binding their custom network to their build environment by leveraging a custom builder. So far everyone who has tried this, including myself, has reported failure. Consider this an open request for a working example. |
This application design is strongly at odds with the general best practices of microservices design and the reason that there is so much friction attempting to implement it within Docker. It implies that the state of the packaged application also includes state in an external and independent database. That tight linkage of state creates issues when upgrading and downgrading container images independent of the database. Specifically for this issue, I'd propose that the DB should not be needed to convert migration steps into a database schema file, and that tooling that could convert between the two would significantly improve the build process for that application. If that's feasibly impossible, then starting a database, running the migration, and dumping the schema, could be done in a single step that runs the database on localhost in the background, eliminating the need for a network. I get that this is counter to the design of some popular application frameworks. The difficulty is they are directly opposed to other efforts in the software development ecosystem, like the 12 Factor App, and now the desire to provide hermetic and reproducible builds.
I posted exactly that back in January in this very issue. What was missing from my example? |
From a distance, your comment from January of this year (#175 (comment)) could be read as a demonstration of the problem, not of a solution. On a closer read, it does explain my issue. If my priorities were such that the ability to use custom networks in the build process were worth the cruft of putting Maybe if this pattern gets wrapped into a Docker Compose plugin, that plugin could also do something like add the container's name and IP to
In this case the starting point is a DB image. The only reason I can't apply migrations within the build script is because the DB image and the migration script image (which expects to be able to connect to a DB over the network) are two separate images, and to coordinate between them in the same dockerfile would require inter-process communication between build stages. Hence instantiating the DB on a custom network as an attempted workaround. However, the migration script image is really just a wrapper around a binary, so I’m realizing my best bet is to copy that binary into my DB image and run it there which cuts out the need for an external network. Good riddance! |
Per rest of my quoted sentence, "both from outside influences and particularly from Docker.". Specifically, the build engine is no longer part of the Docker project, in the same way the container engine stopped being part of Docker and became containerd. This allows it to be useful to and contributed-to by a wider variety of projects and be able to be used in a wider variety of environments. Speculatively, Docker saw the writing on the wall that things like Kubernetes were not going to keep Docker underlying them forever, so anything that required Docker (like the class Docker builder) was going to fall by the wayside. You can see lots of attempts to build non-Docker container build projects, but having tried a fair few back in the day for scaling/distributed builds on k8s, buildkit has been my personal best-result, and you can even drive k8s-based buildkit instances from the Docker CLI etc using buildx, so it's also the least-painful way for users familiar with the Docker ecosystem to access that wider build ecosystem.
Roughly, yes. I guess maybe it'd be possible if someone managed to write a CNI plugin that could hook up an arbitrary container to the Docker libnetwork-driven overlay networks on the various platforms, and then work out how to bundle that inside the buildx or dockerd binary (which CNI probably makes impossible, as it relies on command-line interfaces and special binary name/paths to function). However, it's possible (I haven't looked closely in a while) that the in-Docker buildkit instance can't use CNI with its OCI work for some other structural reason, even if such a CNI plugin was available. Honestly, the simplest code-solution to make Just to be clear, the decision wasn't made by BuildKit. BuildKit is one of the results of those decisions.
I meant this in the sense that it used to be a single command-line or Compose-file option, and now you have to do approximately three commands to make it work. As seen by the years of comments on this and related tickets, that's clearly a significant UX difference. In my opinion, anything you can bundle up into a config file that drives a whole ecosystem of tools is "magic" compared to having to maintain shell scripts, and I don't think it's strange to say so.
I've posted two working examples in this ticket just this year: One with And a quick check shows approximately 0 follow-up comments reporting failure to use those approaches. |
|
Thanks @TBBle for the added context, I feel quite clear about this now. The inability to leverage DNS for custom networks in the build environment is the final hurdle that might have tripped folks up. I’ve seen some failure reports somewhere, maybe not in this issue but somewhere in the 2 o'clock haze from last night. Presumably they didn’t try your example because that does make the solution quite explicit.
I am in favour of this as a solution, particularly if it can somehow solve the DNS issue. I realize the solution I posted about above (https://github.com/docker/buildx/issues/175#issuecomment-2379790250)—modifying the /etc/hosts file for the build container—would have serious problems (not robust to container’s operating system, invalidates build cache layers each time a container or network is rebuilt) |
|
Which DNS issues do you mean? There was a long-standing bug with DNS access to the linked services due to BuildKit's resolv.conf mangling needing an opt-out for host-networking mode, but that was fixed in February and was shipped in BuildKit 0.13. That's why I was using Edit: Also, if you are looking at my examples, passing |
|
@TBBle I was referring to the hostname-based lookup issue. I see now that is fixed. Thank you. The command I tried was |
and others
Background:
Running a simple integration test fails with network option:
docker network create custom_networkdocker run -d --network custom_network --name mongo mongo:3.6docker buildx build --network custom_network --target=test .Output:
network mode "custom_network" not supported by buildkitStill not supported? Code related:
https://github.com/docker/buildx/blob/master/build/build.go#L462-L463
The text was updated successfully, but these errors were encountered: