Remote building through SSH is hanging without explicit `--driver=docker-container` option

[siketyan]

Description

Building images through SSH using remote driver is not working now, since v0.9.0.
BuildX hangs after calling docker buildx build command with no stdout/stderr output.

Remote building hangs after calling docker buildx build or docker buildx create --bootstrap until passing --driver=docker-container option explicitly, since v0.9.0.

According to @jedevc this is because BuildX waits until the server accepts connections to buildkitd instance.

I created a builder using the command line below:

docker buildx create --bootstrap --name ec2 --platform linux/arm64 ssh://siketyan@[redacted].ap-northeast-1.compute.amazonaws.com

Affected versions

v0.9.0 or newer (confirmed on both v0.9.0 and v0.9.1)

Environment

Ubuntu 20.04 LTS (amd64)

[jedevc]

Ah, we recently changed the default driver detection logic in #1134: see https://github.com/docker/buildx/pull/1134/files#diff-577764d60d85606436fc71c2393446d6853531249d5fc9a191594feda5a1ad63R9.

Essentially, the command line you're using is attempting to connect to a raw port on your ssh target, attempting to find an instance of buildkitd listening there. Because --bootstrap is passed, the command is hanging until it can establish a connection.

To avoid the ambiguity, you can pass --driver=docker-container as a flag to force the previous behavior.

[siketyan]

@jedevc
It worked by passing --driver=docker-container, thank you for your advice!

By the way I think this hanging behaviour is still a problem, and am going to change this issue to focus that. If you think that is a specification, not an issue, please close this issue :)

[jedevc]

The reason it hangs is because the behavior of the remote driver when passed --bootstrap is to wait until it can successfully establish a connection. This is useful in things like CI or scripts, where you might want to explicitly wait until a builder is ready before moving onwards to the next step.

We should leave this issue open for the moment though, @tonistiigi suggested we might want to revert to the previous behavior (though I think it might invalidate #1134), so we can track that here.

[nikelborm]

Hi, @siketyan
Do you remember by any chance, did docker buildx create hang for you with the message similar to the one below?

[+] Building 20.0s (1/1) FINISHED                                                                                                                             
 => ERROR [internal] waiting for connection             20.0s
------
 > [internal] waiting for connection:
------
ERROR: context deadline exceeded

I think this issue may be related to moby/buildkit#4327 and moby/buildkit#4268 or even be a duplicate?

[siketyan]

@nikelborm Sorry, I do not remember anything about this issue 😢
I'll close the issue per I don't experience this anymore.

[trajano]

@nikelborm I think you're talking about the same experience as I did here moby/buildkit#4268 (comment)