Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor build/controller API to use single client.Build call #1750

Closed
wants to merge 4 commits into from

Conversation

jedevc
Copy link
Collaborator

@jedevc jedevc commented Apr 19, 2023

Depends on #1737.

This PR performs some refactoring after #1640.

To summarize:

  • Recombine Build and BuildWithResultHandler together. They both return partially evaluated results in ResultContext. These results must be Waited to get the actual build response.
  • Only use a single client.Build call. We only need one gateway session.
  • Fix multi-platform and --print builds to work over the controller interface.
  • Fix --print and --metadata-file logic to happen on the client side.
  • Fix the staggered progress noticed in monitor: add debug-shell and on-error #1640 (comment)
    • This was the main motivator for using a single client.Build session. If we compute the result across two gateway sessions, then we either have to ignore the incoming progress and print out only the final progress results, or we have to merge the two progresses together to display in a single terminal (which... doesn't go well, we get a lot of duplicates).
  • Change the controller API: Build only performs the parts of the build relevant to actually evaluating the result, to perform the export, Finalize must be called with the targeted build ref.
    • There's a good reason for this change - to get all of the invocation progress into a single Build call to BuildKit, we have to perform all invoke calls before we perform an export. That's because, to trigger the export, we have to close the Build call, which would then leave us unable to invoke in the future using that Build.
    • This is actually a good thing 🎉 This means that we won't perform exports before invoke is called, which we were having to do before.
    • The downside here is we have to work out how to "pause" the progress output while we're in monitor mode, so that we don't end up accidentally overflowing into the monitor - not quite sure the best way to do it, I've added a hack into buildkit vendor.

jedevc added 4 commits April 19, 2023 14:44
Refactor the progress printer creation to the caller-side of the
controller api. Then, instead of passing around status channels, we can
simply pass around the higher level interface progress.Writer, which
allows us to correctly pull warnings out of the remote controller.

Signed-off-by: Justin Chadwell <me@jedevc.com>
Signed-off-by: Justin Chadwell <me@jedevc.com>
Signed-off-by: Justin Chadwell <me@jedevc.com>
Signed-off-by: Justin Chadwell <me@jedevc.com>
@jedevc jedevc requested review from tonistiigi and ktock April 19, 2023 17:27
@jedevc jedevc changed the title Refactor build/controller API to only allow single Refactor build/controller API to use single client.Build call Apr 19, 2023
Copy link
Collaborator

@ktock ktock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

if err1 := printer.Wait(); err == nil {
err = err1
res, err := cbuild.RunBuild(ctx, dockerCli, opts, os.Stdin, printer)
resp, err2 := res.Wait(ctx)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should check res != nil to avoid panic

@@ -131,7 +40,39 @@ type ResultContext struct {
cleanupsMu sync.Mutex
}

func (r *ResultContext) Done() {
func (r *ResultContext) hook(hook func(ctx context.Context) error) {
r.hooks = append(r.hooks, hook)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like called from goroutines so should be protected by lock.

Comment on lines -148 to +144
ref, _, err := c.Build(ctx, *bo, nil, stdout, stderr, progressMode) // TODO: support stdin, hold build ref
ref, err := c.Build(ctx, *bo, nil, progress) // TODO: support stdin, hold build ref
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that no progress is printed on reload?

@jedevc
Copy link
Collaborator Author

jedevc commented May 26, 2023

Closing in preference of #1804.

@jedevc jedevc closed this May 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants