Add buildx build --opt (EXPERIMENTAL)

[AkihiroSuda]

This allows opt-in to new features of BuildKit that are not officially exposed to buildx yet.

[crazy-max]

Seems fine if we keep this flag under experimental. Don't think we should move it out in the future?

This could also be useful for bake (can be in follow-up).

If you could add a basic integration test in https://github.com/docker/buildx/blob/master/tests/build.go that checks a frontend attribute being set through opt. Maybe similar to

buildx/tests/build.go

Line 325 in 97e8fa7

func testBuildAnnotations(t *testing.T, sb integration.Sandbox) {

that would set an annotation that overrides one set by --annotation?

[AkihiroSuda]

Added tests and bake.

[AkihiroSuda]

Failure seems unrelated?
https://github.com/docker/buildx/actions/runs/7975611949/job/21774316421?pr=2260

#29 [validate-upstream 5/5] RUN htmltest
#29 0.066 htmltest started at 03:09:57 on public
#29 0.066 ========================================================================
#29 6.066 engine/reference/commandline/buildx_imagetools/index.html
#29 6.066   target does not exist --- engine/reference/commandline/buildx_imagetools/index.html --> /reference/cli/docker/buildx/#builder
#29 6.066 engine/reference/commandline/buildx_create/index.html
#29 6.081   target does not exist --- engine/reference/commandline/buildx_create/index.html --> /reference/cli/docker/buildx/build/#load
#29 6.081   target does not exist --- engine/reference/commandline/buildx_create/index.html --> /reference/cli/docker/buildx/build/#load
#29 6.081   target does not exist --- engine/reference/commandline/buildx_create/index.html --> /reference/cli/docker/buildx/build/#load
#29 6.081   target does not exist --- engine/reference/commandline/buildx_create/index.html --> /reference/cli/docker/buildx/build/#load
#29 6.081 engine/reference/commandline/buildx_inspect/index.html
#29 6.111   target does not exist --- engine/reference/commandline/buildx_inspect/index.html --> /reference/cli/docker/buildx/#builder
#29 6.111 engine/reference/commandline/buildx_use/index.html
#29 6.[352](https://github.com/docker/buildx/actions/runs/7975611949/job/21774316421?pr=2260#step:6:358)   target does not exist --- engine/reference/commandline/buildx_use/index.html --> /reference/cli/docker/buildx/#builder
#29 6.352 engine/reference/commandline/buildx_debug_build/index.html
#29 6.383   target does not exist --- engine/reference/commandline/buildx_debug_build/index.html --> /reference/cli/docker/image/build/#add-host
#29 6.383   target does not exist --- engine/reference/commandline/buildx_debug_build/index.html --> /reference/cli/docker/image/build/#cgroup-parent
#29 6.383   target does not exist --- engine/reference/commandline/buildx_debug_build/index.html --> /reference/cli/docker/image/build/#file
#29 6.383   target does not exist --- engine/reference/commandline/buildx_debug_build/index.html --> /reference/cli/docker/image/build/#tag
#29 6.383   target does not exist --- engine/reference/commandline/buildx_debug_build/index.html --> /reference/cli/docker/image/build/#target
#29 6.383 engine/reference/commandline/buildx_rm/index.html
#29 6.558   target does not exist --- engine/reference/commandline/buildx_rm/index.html --> /reference/cli/docker/buildx/#builder
#29 6.558 engine/reference/commandline/buildx_imagetools_create/index.html
#29 6.618   target does not exist --- engine/reference/commandline/buildx_imagetools_create/index.html --> /reference/cli/docker/buildx/#builder
#29 6.618 engine/reference/commandline/buildx_stop/index.html
#29 6.642   target does not exist --- engine/reference/commandline/buildx_stop/index.html --> /reference/cli/docker/buildx/#builder
#29 6.642 engine/reference/commandline/buildx_prune/index.html
#29 6.880   target does not exist --- engine/reference/commandline/buildx_prune/index.html --> /reference/cli/docker/buildx/#builder
#29 6.880 engine/reference/commandline/buildx_bake/index.html
#29 7.560   target does not exist --- engine/reference/commandline/buildx_bake/index.html --> /reference/cli/docker/buildx/#builder
#29 7.560   target does not exist --- engine/reference/commandline/buildx_bake/index.html --> /reference/cli/docker/buildx/build/#progress
#29 7.560   target does not exist --- engine/reference/commandline/buildx_bake/index.html --> /reference/cli/docker/buildx/build/#provenance
#29 7.560   target does not exist --- engine/reference/commandline/buildx_bake/index.html --> /reference/cli/docker/buildx/build/#sbom
#29 7.560 engine/reference/commandline/buildx_imagetools_inspect/index.html
#29 7.716   target does not exist --- engine/reference/commandline/buildx_imagetools_inspect/index.html --> /reference/cli/docker/buildx/#builder
#29 7.716 engine/reference/commandline/buildx_build/index.html
#29 8.107   target does not exist --- engine/reference/commandline/buildx_build/index.html --> /reference/cli/docker/image/build/
#29 8.107   target does not exist --- engine/reference/commandline/buildx_build/index.html --> /reference/cli/docker/image/build/#add-host
#29 8.107   target does not exist --- engine/reference/commandline/buildx_build/index.html --> /reference/cli/docker/image/build/#cgroup-parent
#29 8.107   target does not exist --- engine/reference/commandline/buildx_build/index.html --> /reference/cli/docker/image/build/#file
#29 8.107   target does not exist --- engine/reference/commandline/buildx_build/index.html --> /reference/cli/docker/image/build/#tag
#29 8.107   target does not exist --- engine/reference/commandline/buildx_build/index.html --> /reference/cli/docker/image/build/#target
#29 8.107   target does not exist --- engine/reference/commandline/buildx_build/index.html --> /reference/dockerfile/#run---securitysandbox
#29 8.107   target does not exist --- engine/reference/commandline/buildx_build/index.html --> /reference/cli/docker/buildx/create/#buildkitd-flags
#29 8.107   target does not exist --- engine/reference/commandline/buildx_build/index.html --> /reference/cli/docker/image/build/#build-arg
#29 8.107   target does not exist --- engine/reference/commandline/buildx_build/index.html --> /reference/dockerfile/#buildkit-built-in-build-args
#29 8.107   target does not exist --- engine/reference/commandline/buildx_build/index.html --> /reference/cli/docker/buildx/#builder
#29 8.107   target does not exist --- engine/reference/commandline/buildx_build/index.html --> /reference/dockerfile/#automatic-platform-args-in-the-global-scope
#29 8.107   target does not exist --- engine/reference/commandline/buildx_build/index.html --> /reference/dockerfile/#run---mounttypesecret
#29 8.107   target does not exist --- engine/reference/commandline/buildx_build/index.html --> /reference/dockerfile/#run---mounttypessh
#29 8.107 ========================================================================
#29 10.51 ✘✘✘ failed in 10.442016513s
#29 10.51 35 errors in 2083 documents
#29 10.51 
#29 ERROR: process "/bin/sh -c htmltest" did not complete successfully: exit code: 1

[crazy-max]

Failure seems unrelated? https://github.com/docker/buildx/actions/runs/7975611949/job/21774316421?pr=2260

Yes this is linked to #2253 and will be fixed when changes are made on docs repo (cc @dvdksn)

[tonistiigi]

I'm not sure if we want this. Buildx uses opinionated UX and strictly defined flags/types. Allowing to set raw buildkit values directly adds confusion and makes it easier to make mistakes - pass wrong values and get to undefined configuration combinations. Definitely not as --opt as the meaning is more "raw buildkit frontend config option".

[AkihiroSuda]

I'm not sure if we want this. Buildx uses opinionated UX and strictly defined flags/types. Allowing to set raw buildkit values directly adds confusion and makes it easier to make mistakes - pass wrong values and get to undefined configuration combinations. Definitely not as --opt as the meaning is more "raw buildkit frontend config option".

This will remain experimental forever, so this will not confuse non-expert users.

If this is still not acceptable as an arg, can we implement this as an env var that takes a JSON of []string?

[colinhemmings]

Hi there, I agree with Tonis on this one. I'm concerned about the user experience of enabling this kind of backdoor through to Buildkit. It's confusing for users and leads to a support overhead further down the line. I know this would be experimental, but people would start using it and then get annoyed when we removed it or stopped supporting it. We already get backlash from users regarding legacy builder functionality, which has been dropped from Buildkit.
If there are specific features/use cases that are not available via Buildx, we should consider and implement those. That way, all users benefit from them with proper tested support.