kubernetes driver fails to configure multi-node setup due to missing node option
#215
Comments
|
I think the following should work: - name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
with:
driver: kubernetes
platforms: linux/amd64
driver-opts: |
namespace=buildkit
nodeselector=kubernetes.io/arch=amd64
append: |
- platforms: linux/arm64
driver-opts:
- namespace=buildkit
- nodeselector=kubernetes.io/arch=arm64I will open a PR to set up k3s in our ci to see how it goes. |
|
without |
Inspecting this builder: {
"nodes": [
{
"name": "builder-c914f57b-bb09-40ec-9887-0b2e371f89e70",
"endpoint": "unix:///var/run/docker.sock",
"status": "running",
"buildkitd-flags": "--allow-insecure-entitlement security.insecure --allow-insecure-entitlement network.host",
"buildkit": "v0.11.4",
"platforms": "linux/amd64,linux/amd64/v2,linux/amd64/v3,linux/amd64/v4,linux/386"
},
{
"name": "builder2",
"endpoint": "ctxbuilder2",
"driver-opts": [
"image=moby/buildkit:master",
"network=host"
],
"status": "running",
"buildkit": "252e5a6",
"platforms": "linux/amd64"
},
{
"name": "builder-c914f57b-bb09-40ec-9887-0b2e371f89e71",
"endpoint": "ctxbuilder3",
"status": "running",
"buildkit": "v0.11.4",
"platforms": "linux/arm64"
}
],
"name": "builder-c914f57b-bb09-40ec-9887-0b2e371f89e7",
"driver": "docker-container",
"lastActivity": "2023-03-10T10:04:38.000Z"
}Can you show the full logs of the |
|
Thanks for looking into this. I could repro this with one of our public repos. Please see https://github.com/overmindtech/aws-source/actions/runs/4385572203/jobs/7678475151#step:7:128 (raw log |
|
I see in ci.yml that it is defining both builds in |
|
Indeed with both builders in |
|
Looking at our docs https://docs.docker.com/build/drivers/kubernetes/#native it seems you effectively need to define Also it seems Can you try with: - name: Set up Docker Buildx
id: setup-builder
uses: docker/setup-buildx-action@v2
with:
driver: kubernetes
platforms: linux/amd64
driver-opts: |
namespace=buildkit,nodeselector="kubernetes.io/arch=amd64"
append: |
- name: arm64
platforms: linux/arm64
driver-opts:
- namespace=buildkit,nodeselector="kubernetes.io/arch=arm64" |
Does not seem to work properly: https://github.com/overmindtech/aws-source/actions/runs/4385745527
Hence my previous comment about having a single |
|
nice! https://github.com/overmindtech/aws-source/actions/runs/4385908114/jobs/7679252628 built and cleaned up |
Yes looks good! {
"nodes": [
{
"name": "builder-a02cb880-5b61-445e-a2ae-98d6f3e2283f0",
"endpoint": "kubernetes:///builder-a02cb880-5b61-445e-a2ae-98d6f3e2283f?deployment=&kubeconfig=",
"driver-opts": [
"namespace=buildkit",
"nodeselector=kubernetes.io/arch=amd64"
],
"status": "running",
"buildkitd-flags": "--allow-insecure-entitlement security.insecure --allow-insecure-entitlement network.host",
"buildkit": "v0.11.4",
"platforms": "linux/amd64"
},
{
"name": "arm64",
"endpoint": "kubernetes:///builder-a02cb880-5b61-445e-a2ae-98d6f3e2283f?deployment=arm64&kubeconfig=",
"driver-opts": [
"namespace=buildkit",
"nodeselector=kubernetes.io/arch=arm64"
],
"status": "running",
"buildkit": "v0.11.4",
"platforms": "linux/arm64"
}
],
"name": "builder-a02cb880-5b61-445e-a2ae-98d6f3e2283f",
"driver": "kubernetes",
"lastActivity": "2023-03-10T15:20:44.000Z"
} |
|
whoops, had the |
|
yeah, https://github.com/overmindtech/aws-source/actions/runs/4385935559/jobs/7679313927 failed with the same At least now I have a working config - I can specify the name from somewhere unique, like adding |
You could use - name: Set up Docker Buildx
id: setup-builder
uses: docker/setup-buildx-action@v2
with:
driver: kubernetes
platforms: linux/amd64
driver-opts: |
namespace=buildkit,nodeselector="kubernetes.io/arch=amd64"
append: |
- name: run-${{ github.run_id }}-arm64
platforms: linux/arm64
driver-opts:
- namespace=buildkit,nodeselector="kubernetes.io/arch=arm64"I wonder if we should do this automatically with the action when the kubernetes driver is being used. WDYT? |
|
Reading through the docs, I think this would need both to avoid any clashes with concurrent runs (although I don't know if multiple attempts of the same run can run concurrently):
so this works for me: commit, run. It still also needs to consider the job and a few other things that could clash. really, it would be nice if the action could just chuck in a UUID and not have me think about these things :-D |
|
e.g. using |
Yeah we could do the same thing as: setup-buildx-action/src/context.ts Line 45 in 4b4e9c3 |
|
What I found strange is the generated node name should not collide with anything. For example in our ci workflow: setup-buildx-action/.github/workflows/ci.yml Lines 435 to 448 in 4b4e9c3 For this run builder name is
So for me it should be fine is you don't define any name for the nodes as it takes the generated builder name and append the node position ( |
|
those are on two different |
Ah good point that might be it. I will add a new job in our ci to check this out. |
|
@DavidS-om #219 should fix this issue. See https://github.com/docker/setup-buildx-action/actions/runs/4387426773/jobs/7682713422#step:4:171 You can test with |
|
Awesome. I will open a follow-up to fix this issue upstream on buildx repo. |
Behaviour
Steps to reproduce this issue
Expected behaviour
Create a working builder with an arm64 and an amd64 node
Actual behaviour
Workaround
this seems to be working:
The text was updated successfully, but these errors were encountered: