[archived] Docker build inside Codespaces used to confuse Dockerfiles unless you run touch Dockerfile first

[michielbdejong]

UPDATE

This discussion was about a bug in Docker that manifested itself in GitHub Codespaces in 2023, but is now fixed and resolved.

Select Topic Area

Bug

Body

See https://github.com/michielbdejong/codespaces-weirdness - the weirdness is explained in the readme there.

[michielbdejong]

I updated the title to be more descriptive of the actual bug.
Is this the right place for Codespaces beta testers to report bugs? Is this getting picked up by the Codespaces dev team?

[michielbdejong]

Bump

Is this the right place for Codespaces beta testers to report bugs?

So far, apparently not - any tips about how to reach them would also be appreciated!

[michielbdejong]

See answer below:

Hi from the Codespaces team 👋. Thanks for reporting this and providing a very clear minimal repro of the problem. I took a look, and this is a bug is Docker/Moby's BuildKit engine: moby/buildkit#1368. I don't think it's a codespaces-specific issue, but Codespace does enable BuildKit by default, which could explain why it happens in Codespaces.

Disabling the DOCKER_BUILDKIT environment variable is a good workaround (e.g. DOCKER_BUILDKIT=0 docker build -t d2 . in your example repo). You could also set a user-level or repo-level secret to set that environment variable on multiple codespaces.

[michielbdejong]

Hm, no reply there either yet...

[michielbdejong]

Ah, they replied, https://support.github.com/ticket/personal/0/1883475 is now closed, but it has been opened in a third issue tracker:

In order to remain proactive and keep our ticket queue manageable, we cannot keep tickets open without activity for extended periods of time. I'm afraid our system is also going to mark this ticket as "solved" now that the issue has been escalated with the engineering and product team, but that doesn't mean we forget about it!

[jkeech]

Hi from the Codespaces team 👋. Thanks for reporting this and providing a very clear minimal repro of the problem. I took a look, and this is a bug is Docker/Moby's BuildKit engine: moby/buildkit#1368. I don't think it's a codespaces-specific issue, but Codespace does enable BuildKit by default, which could explain why it happens in Codespaces.

Disabling the DOCKER_BUILDKIT environment variable is a good workaround (e.g. DOCKER_BUILDKIT=0 docker build -t d2 . in your example repo). You could also set a user-level or repo-level secret to set that environment variable on multiple codespaces.

[jkeech]

I also confirmed that the same issue reproduces in GitPod and other locations when BuildKit is enabled.

[michielbdejong]

Thanks a lot for digging into this and finding the cause!
My trust in Codespaces as a usable product is definitely restored by that. :)
Is it an option for you to update your default Docker version to one that contains the fix?
Until then, maybe as a hotfix, set DOCKER_BUILDKIT=0 as the default across your product?
Or at least print a warning message about it each time you run docker build inside Codespaces?
I asked for a recommended course of action in moby/buildkit#1368 (comment)

[jkeech]

The Codespaces default image uses the latest docker/moby and buildx versions and will continue to pick updates up soon after they are published. The exact mechanism for picking up newer builds can be seen (and re-used in your custom devcontainer definitions) in the docker-in-docker devcontainer feature: https://github.com/devcontainers/features/tree/main/src/docker-in-docker.

Unfortunately this bug doesn't appear to be fixed in the latest versions of Moby/buildx yet which is why it's still occurring. Overall, BuildKit brings a lot of improvements to performance and the overall experience of working with docker containers locally, and while this is a bad bug, we believe keeping BuildKit enabled by default is the right experience for the vast majority of Codespaces users and would recommend disabling it as needed for specific repos.

[michielbdejong]

This is reportedly fixed in Docker v23 and it looks like CodeSpaces now has v24 so I'll re-test https://github.com/michielbdejong/codespaces-weirdness now.

[michielbdejong]

Confirmed, this no longer occurs in GitHub CodeSpaces today! \o/