[FEATURE]: Support buildkit output compression arguments

[alexkuretz]

Feature Request

Allow configuration of buildkit output compression options such as zstd
https://github.com/moby/buildkit?tab=readme-ov-file#building-a-dockerfile-with-buildctl

Background / Motivation

This could dramatically speed up pod deployment times while decreasing image size.
https://aws.amazon.com/blogs/containers/reducing-aws-fargate-startup-times-with-zstd-compressed-container-images/

Currently the output is hardcoded I tried setting this through spec.extraArgs but that just creates duplicate outputs and didn't work:

extraFlags: ["--output", 'type=image,compression=zstd,force-compression=true,compression-level=3']

{"msg":"Execing command in my-namespace/Pod/garden-buildkit-6f9f66dc4f-tkl2w/buildkitd: sh -c cd /garden-build/ade67c77-f2b6-458e-b102-23d8ebbd7414/my-module 
&& 'buildctl' 'build' '--frontend=dockerfile.v0' 
'--local' 'context=/garden-build/ade67c77-f2b6-458e-b102-23d8ebbd7414/my-module' 
'--local' 'dockerfile=/garden-build/ade67c77-f2b6-458e-b102-23d8ebbd7414/my-module' 
'--opt' 'filename=Dockerfile' 
'--output' 'type=image,\"name=1234567890.dkr.ecr.us-west-2.amazonaws.com/my-repo/my-module:v-b72b75b844\",push=true' 

<snip>

 '--output' 'type=image,compression=zstd,force-compression=true,compression-level=3'","section":"build.my-module             →","timestamp":"2024-10-30T22:28:01.748Z","level":"debug"}

Build seemed successful and image was smaller, but got an error on deploy:

10m (x4 over 12m)      Warning   Failed              Pod/my-module-67dfc7d49d-96ksq            Failed to pull image
"1234567890.dkr.ecr.us-west-2.amazonaws.com/my-namespace/my-module:v-b72b75b844": failed to pull and unpack image
"1234567890.dkr.ecr.us-west-2.amazonaws.com/my-namespace/my-module:v-b72b75b844": number of layers and diffIDs don't match: 26 != 51

What should the user be able to do?

Define compression options on the buildkit output

Why do they want to do this? What problem does it solve?

Faster pod spinup, smaller images, multithreaded compression, probably more

Suggested Implementation(s)

Maybe add spec.outputOptions that get pushed onto the output command?

How important is this feature for you/your team?

🌵 Not having this feature makes using Garden painful
somewhere in here
🌹 It’s a nice to have, but nice things are nice 🙂

[stefreak]

Thank you for the valid feature request 👍