BuildKit builds (via Docker) are broken if /etc/hosts or /etc/resolv.conf is replaced #1267
Comments
|
What version of docker is this? |
|
I can give you an exact version when I'm at the console again, but it's whatever the current version for Ubuntu is (Docker's repositories, not Ubuntu's). I've also reproduced the problem with a current |
|
@tonistiigi my version is |
|
Yes, this is somewhat expected by the current implementation. These files are configured by the container runtime. Even with the old implementation, these files do not really persist, you can write into them but the next command will not see the data you wrote, leading to a different error case. BuildKit will not allow writes instead of silently ignoring them. We could try to keep compatibility but it does have a bit of a performance overhead that would apply to all the containers, not just the ones the try to write to these files. |
|
I edited them to fake the docker hostname for applications that rely on these files and it helped and needed just during build stage due the limit to change the host. |
|
I was having this issue, under the impression that |
|
Hi, If you want to set specific hostname for during build time, now it implemented under this pull request: |
|
Also running into this... Is there any way to allow It's okay if the I tried PS: this older issue is not really related: moby/moby#11950 |
|
I am also running into this, just trying to build a simple archlinux container. Turning off buildkit solves this problem. It would appear recently (sometime, in 2020 maybe, between June and December?) Docker for Mac made a signfigant shift to build kit? This same Dockerfile that used to work now does not. I posted this question before finding this thread, for full details: https://unix.stackexchange.com/questions/631177/arch-linux-in-docker-on-a-free-system-is-running-out-of-space/631178#631178 |
|
I don't think there are plans to "fix" this. As I explained before these files are not modifiable in containers and managed by the runtime. Changing them leads to confusing behavior and broken images. |
|
I understand. But I think what will happen is that people are going to hit this and it will be "obscure" to them because it's indirect. I'm not directly doing anything with resolve or hosts. I'm merely building the same docker image that I've built for years, in the same way, and now suddenly it doesn't work. The error that gets posted out, as I wrote in that stackexchange post highlights the error here, albeit with a minor red herring about space, but again to the user it's Indirect unless they are doing things (purposefully) with the hosts or resolve file. People running say just an OS like Arch or Ubuntu, this could be a "hidden" landmine in their typical work flow; especially since they may not be aware that common docker tools have recently switched to buildkit "under the covers" (docker for mac for example) |
It's a tricky one; I agree that for users that want to customise during build, I saw moby/moby#11950 was linked above; probably relevant here are moby/moby#2267, and moby/moby#5129, which (although not "persisting"), makes both of these writable by mounting the files. I do share the concerns about performance though, so not sure what's best. |
|
@thaJeztah sorry to labour the issue, but am i right in saying that there is still no workaround for the original issue, where installing arch linux package 'filesystem' causes a read only error due to bind mounts to the host?, i have gone through the links above but cannot see a way around this issue that relate to installation of a package that needs write access to /etc/hosts and /etc/resolv.conf please correct me if i'm wrong here. |
|
@binhex correct; BuildKit currently mounts these read-only; With BuildKit: DOCKER_BUILDKIT=1 docker build --no-cache --progress=plain -<<EOF
FROM busybox
RUN mount | grep /etc
RUN cat /etc/hosts
EOFWithout BuildKit: DOCKER_BUILDKIT=0 docker build --no-cache --progress=plain -<<EOF
FROM busybox
RUN mount | grep /etc
EOFIf your intent is to install the |
|
Just for anybody who finds this useful, i stumbled across the OP issue when using GitHub Actions and attempting to build a Arch Linux Docker image 'from scratch' using the buildx step:- if i then used the following 'uses' action then it blew up with the same read only issue for the solution for me is simply to ignore the filesystem package during upgrade by doing the following:- pacman then happily ignores the upgrade of filesystem and completes, this is obviously not ideal but it works and im happy enough to share this as a workaround for now until when/if the |
Since we started building the nginx-tls image in the pipeline with the
vito/oci-build-task image, the `build-nginx-tls` Concourse job has been
failing.
The build attempts to change the owner of the `/etc/hosts` directory to
`nginx`. This fails with the error message:
`chown: /etc/hosts: Read-only file system`
The oci-build-task uses BuildKit directly under the hood, rather than
with Docker. I believe this is causing the issue.
It seems that `/etc/hosts` is treated specially by the build runtime,
and BuildKit does not allow writes to the dir. There is some discussion
of it in [this issue.](moby/buildkit#1267).
It seems that [you need to manually turn BuildKit on in Docker](https://docs.docker.com/develop/develop-images/build_enhancements/), which
would explain why this has only just broken.
To get around this issue, we can change the owner of `/etc/hosts` after
the build, in the entrypoint script. To do this we need to be the root
user, which is why the `USER nginx` instruction has been removed from the
Dockerfile.
Once the owner of `/etc/hosts` has been changed we can switch to the
`nginx` user. To do this from the entrypoint script the nginx users
needs to have a login shell set. That's what the `sed` command is doing.
I'm not 100% sure that this is the right approach for this. I'm not sure
why the `/etc/hosts` dir needs to be owned by nginx in the first place,
but it's probably been done for a good reason. But the simplest fix
would be to not change the ownership at all.
|
A lot of posts here mention Is there any way to explicitly set nameservers for a single |
Runners use Docker v24 now, which uses BuildKit by default. This broke the build when reinstalling `filesystem` due to moby/buildkit#1267 Also expand out the package list in the Dockerfile for easier diffs when adding/removing packages.
I understand the situation to use --add-host for docker build, but how to use it for buildctl? I couldn't find any way to use --add-host in |
|
@TingxinLi |
Thanks, I figured it out after reading the source code. |
|
how can i edit: |
|
I have to add to this issue as well... I either need /etc/hosts to be writable or need the hostname to not resolve to a loopback address. Reasoning: I have to prebuild an Oracle database during a container build, sigh... (With their current free database version 23.5, they don't accept loopback addresses as IPs for the given hostname and this hostname is taken from /etc/hostname, which is also read-only and correlates with the 127.0.0.1 entry in /etc/hosts. Adding the hostname with a different IP again using --add-host is ignored because it gets appended to the end of /etc/hosts. For now, I have to stick to the non-buildkit Docker build mode because it seems easier than getting a fix from Oracle...) |
|
Archlinux issues is still an issue, why this is not configurable in BuildKit is beyond any sane explanation. |
My Dockerfile:
The
filesystempackage contains most of the base system files, as can be seen at https://www.archlinux.org/packages/core/x86_64/filesystem/. If it's upgraded, it tries to write/etc/hostsand/etc/resolv.conf, which leads to errors with BuildKit ("classical" Docker build is fine):The text was updated successfully, but these errors were encountered: