[image-builder] workspace images' environment variables are broken

[davidwindell]

Bug description

Did any deployments or changes to Gitpod take place last night?

Since this morning, none of our workspaces are starting as all of the environment variables (ENV in the Dockerfile) from the base image appear to be being wiped out somehow.

Steps to reproduce

https://gitpod.io/#https://github.com/davidwindell/gitpod-test

Expected behavior

No response

Example repository

Docker image used is https://github.com/outeredge/edge-docker-php-dev/blob/main/Dockerfile.common

For example, echo $PHP_DISPLAY_ERRORS and most importantly echo $NGINX_PORT is empty (from the upstream docker image, causing Nginx to stop booting).

Anything else?

No response

[corneliusludmann]

Thanks for reporting. I can confirm the issue with your sample repo. Thanks a lot for that! I'm currently investigating.

[corneliusludmann]

https://www.gitpodstatus.com/incidents/fk6prflgvvth

[iQQBot]

It seem a bug from moby/buildkit

[corneliusludmann]

It seem a bug from moby/buildkit

@iQQBot Do you have more information about this? Or is it just a guess?

[geropl]

Thanks everybody for reporting (especially @davidwindell), we identified and are rolling have rolled back our changes from this morning.

We are closing this issue now and will keep this one around until the underlying issue is fixed and deployed.

Link to incident: https://www.gitpodstatus.com/incidents/fk6prflgvvth

[corneliusludmann]

We mitigated the issue for now. Please restart a workspace and try again, @davidwindell. Let us know if it does not work.

[kiselev-nikolay]

I had this problem too. I just checked. It looks like you fixed the problem. Thanks, gitpod team!

It would be great if such updates would not interfere with all users. Please add a separate "insiders" mode, which will be used by those who care about the "latest" updates. The work of developers (who entrusted gitpod) depends on you. I would not want to waste time because of such things in the future.

[davidwindell]

@corneliusludmann thank you for the quick resolution, I can confirm this has fixed the issue for our team.