Pipedreams: Buildkit and LLB ideas

[salotz]

This is not a feature request or anything! I just wanted to share some ideas.

As a lot of you are familiar with the new Docker build backend is called Buildkit and it introduces an intermediary representation called LLB and is pretty general purpose beyond just building containers.

For instance the original Docker creator/founder is building a CI/CD platform that runs everything through Buildkit.
Probably more similar to Pants is Earthly, which is a monorepo build system (however with dubious choices in syntax). You then get all the benefits such as caching (and pushing caches and artifacts transparently to standard OCI registries).

I kind of wanted to just stimulate some conversation around if this is something that Pants could adopt as a backend or if there are enough differences in what Pants does that would make this not make sense theoretically.

From what I see there would be a few benefits.

1. Standard artifact (including cache) registry format, protocol, and ecosystem around hosting and accessing them.

Various build systems and package managers all have this kind of M:N problem where they have their own reimplementation of artifact storage and distribution. For instance Python package index. Its a massive, difficult to maintain system with its own quirks and its not practical for anyone to host their own of it. Container registries also have a lot of benefits from sharing images via layers that can make them a lot more efficient. Plus there are numerous options for hosting them and tools for moving those images around.

2. Simplify Pants code base by vendoring a lot of low level code. By this I mean buildkit.
3. Native integration for building containers.