Switch to buildx to build airflow images

[potiuk]

The "buildkit" is much more modern docker build mechanism and supports
multiarchitecture builds which makes it suitable for our future ARM
support, it also has nicer UI and much more sophisticated caching
mechanisms as well as supports better multi-segment builds.

BuildKit has been promoted to official for quite a while and it is
rather stable now. Also we can now install BuildKit Plugin to docker
that add capabilities of building and managin cache using dedicated
builders (previously BuildKit cache was managed using rather
complex external tools).

This gives us an opportunity to vastly
simplify our build scripts, because it has now much more robust caching
mechanism than the old docker build (which forced us to pull images
before using them as cache).

We had a lot of complexity involved in efficient caching
but with BuildKit all that can be vastly simplified and we can
get rid of:

• keeping base python images in our registry
• keeping build segments for prod image in our registry
• keeping manifest images in our registry
• deciding when to pull or pull&build image (not needed now, we can
  always build image with --cache-from and buildkit will pull cached
  layers as needed
• building the image when performing pre-commit (rather than that
  we simply encourage users to rebuild the image via breeze command)
• pulling the images before building
• separate 'build' cache kept in our registry (not needed any more
  as buildkit allows to keep cache for all segments of multi-segmented
  build in a single cache
• the nice animated tty UI of buildkit eliminates the need of manual
  spinner
• and a number of other complexities.

Please take a look only at the last commit as it is based on #20238 #20679

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[potiuk]

This is a VAST simplification of our docker build caching (using modern BuildKit plugin). I have to do some more testing with it, but on top of removing several hundreds of lines of Bash that I implemented when BuildKit did not have all the capabilities needed, it's also very stable and robust and opens the path to multi-architecture images (for ARM).

I would do some more testing - but I would also love to merge #20238 (optimization of Dockerfiles) that this build is based on.

[potiuk]

cc: @Bowrna @edithturn - > that will make your job much easier, it simplifies a lot of the "caching" complexity - that's why I did not want you to start looking at it before as I knew we are going to get it WAAAAY simpler with BuildKit.

[uranusjr]

I’m not too familiar with buildx in the first palce and will dismiss my request for review.

[ashb]

Whats buildx vs buildkit?

[potiuk]

Whats buildx vs buildkit?

Essentially the same. Buildx is a plugin you can install in docker to get more capabilities of buildkit available as "docker buildx build" command (and a number of management commands). You do not need it to run "buildkit-enabled" builds (DOCKER_BUILDKIT=1 is enough in docker 1.18+) but you need it for example to build and push cache to github registry.

An example of that is our prod image. It is multi-segmented image so in order to prepare a good cache for the builds I need to do it with command smilar to docker buildx build . --cache-to=type=registry,ref=ghcr.io/,........,mode=max (mode=max means that the cache includes layers from all segments which means that when you build muiti-segmented build one --cache-from is enough)

This was (about a year ago when I checked last time) missing in buildkit - you had to do some strange combination of not-yet-released-then tools from "moby" to build and refresh the cache- but now with the plugin it's a "breeze" to manage and prepare the cache.

In our case users will not have to install buildx plugin, but it will have to be available (eventually) on our self-hosted runners to refresh the cache on main builds (I will add it) and any time you want to manually refresh the cache with ./breeze prepare-build-cache.

https://docs.docker.com/engine/reference/commandline/buildx_build/

[potiuk]

It also has all the nice things about building multi-platform images. The nice thing about buldx is that you can have mutliple builders - for example different builders for different platforms, or you could have " build cache server" where you builds such caches and organize it in the way that each builder is completely separated from the "docker engine" it runs on. It works in the way that each builder runs as a separate container and has private image 'storage" so when the builder builds an image, it's not visible via docker images.

Pretty nice solution to organize your builds when you have mutli-platform, multi-branch, multil-whatever case. Initially seems much more complex than original docker build system, but it is actually very intuitive.

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[potiuk]

FYI. I will run a few more tests in my fork - just to make sure everything is fine with caching of build-image workflow results and will merge it when I come back from Slovakia (going for a few days) just to make sure to support any problems with Breeze. So still some time for reviews :)