[FEATURE]: Add AWS ECR to supported multi-stage build caching registries for cluster-buildkit

[alexkuretz]

Feature Request

Background / Motivation

AWS and Buildkit now properly support OCI-compliant manifests and layer caching for multi-stage builds. This became available last year in buildkit 0.12.0 and more recently in Docker 25.
aws/containers-roadmap#876

Garden documentation is out of date in this regard.

I tested this by setting mode=max in cluster-buildkit cache settings in project.garden.yml:

    buildMode: cluster-buildkit        
    clusterBuildkit:
      rootless: true
      cache:
        - type: registry
          tag: _buildcache-${camelCase(git.branch)}
          mode: max
          export: false
        - type: registry
          tag: _buildcache-main
          mode: max
          export: false

and setting extraFlags in garden.yml:

extraFlags: ["--export-cache", "image-manifest=true,type=registry,ref=accountnumber.dkr.ecr.us-west-2.amazonaws.com/myproject/myservice:_buildcache-main,mode=max"]

#27 writing cache image manifest sha256:cdb42224716cc1392ebbe44e6d9075f6a84cfc758b79f431c55a96ea5c285128 0.2s done

And in the "Artifact Type detail" for the _buildcache-main image in ECR:

Artifact type detail
Image manifest media type
application/vnd.oci.image.manifest.v1+json
Artifact media type
application/vnd.buildkit.cacheconfig.v0

Some initial quick testing showed an moderate decrease in build times. Between clean tests I deleted images from ECR.
Default cluster-buildkit caching:
Initial clean build took 16m 1s, changing one line in one source file and rebuilding took 12m 9s.

cluster-buildkit with mode=max and image-manifest=true:
Initial clean build took 17m 23s, same one-line change and rebuilding took 9m 42s.

Related issues:
#3110
#3219

What should the user be able to do?

Enable mode=max and set image-manifest=true

Why do they want to do this? What problem does it solve?

Speed up builds by caching layers in multi-stage builds

Suggested Implementation(s)

Suggest adding AWS ECR to the garden list of supported registries, setting mode=max by default and add image-manifest=true to the --export-cache options

How important is this feature for you/your team?

🌵 Not having this feature makes using Garden painful

[stefreak]

@alexkuretz thank you for reporting this! We'll look into it.

[stefreak]

@alexkuretz I implemented AWS ECR mode=max support in #5758

Thanks for letting us know!

By the way, you should be able to save even more time in pull requests by only exporting cache manifests on main, e.g. like this:

      cache:
        - type: registry
          tag: _buildcache-main
          # Only export cache if the workflow runs on the main branch, as exporting the cache can take a lot of time.
          export: ${git.branch == "main"}

This also helps avoiding pollution of the cache by random changes in PRs.

[stefreak]

@alexkuretz would be happy to hear if this works for you. You can try it out by updating to edge-bonsai using the command garden self-update edge-bonsai. Thank you!

[alexkuretz]

Thanks for adding this, it does appear to be working, we're seeing manifests show up in ECR now instead of just images.

[alexkuretz]

I'll add that as we've updated projects to use the new caching we've seen some strange build failures. Deleting the old build_* cache images in ECR seems to resolve it.