Make the use of BuildKit possible in the image building step.

[FeryET]

Hi.

Docker buildkit helps me to build images much faster on my local PC, wanted to see if there is anyway to use the buildkit in the image building phase in prefect_docker?

I can help with the code if needed.

[ngriffiths13]

I second this. It would be great if there was a way to use buildkit.

[desertaxle]

Based on docker/docker-py#2230 it looks like docker-py (the Docker library that prefect-docker uses) does not support BuildKit. As a result, supporting BuildKit in prefect-docker will be difficult. One possible solution is to use another library like pyton-on-whales that supports BuildKit, but migrating will require rewriting large portions of prefect-docker.

[desertaxle]

It looks like it is possible to set the buildkit flag at the daemon level: #10331 (comment).

[FeryET]

I don't think this necessarily address the issue espeically in the code. Since DOCKER_BUILDKIT=1 uses multithreaded build system and caching which AFAIK docker-py does not support. So regardless of the flag, I think it will not work.

I think supporting both Python On Whales and docker-py can be a good option. Since python-on-whales also supports podman containers, and most users do not want fine-grained control over what docker-daemon is doing like what Docker-py offers.

I can help with implementing some flows (building, pulling or pushing) using python-on-whales. Ping me if you thought this was a good idea. I still think for creating, running and managing containers docker-py is more than good enough, just actions like multithreaded pushing, pulling or building are needed.

[ConstantinoSchillebeeckx]

Another reason to use buildkit is to be able to use secrets; as it stands now, we use build args to pass in secrets which are visible in the image history.