Error: headless task failed, on imagebuild

[axonasif]

Bug description

I was trying to open this repo https://gitpod.io/#https://github.com/axonasif/wrover-experimental/tree/wokwi-test after adding a custom dockerfile

Steps to reproduce

Open https://github.com/axonasif/wrover-experimental/tree/wokwi-test on Gitpod.

Workspace affected

axonasif-wroverexperime-u9ik2sre8pl

Expected behavior

No response

Example repository

https://github.com/axonasif/wrover-experimental/tree/wokwi-test

Anything else?

A similar report on Discord today: https://discord.com/channels/816244985187008514/972590280446476308/973228246407733288

[mteam88]

Another issue is that there is no build output other than error text. I had a build run that was successful but had no output.

[ChamMach]

+1, same for me (with a private project of mine that had a Dockerfile update) 🤔

[felipealencar]

Same problem here.

[ChamMach]

Very surprised that there aren't more people complaining about it. While it's all about viewing the outputs/logs, it gets hard not knowing exactly where the problem is during a build with a "lot" of changes.

[felipealencar]

Hello everyone, Idk who still is facing this issue, but I could see detailed information about my error by:

1. Selecting the option to run a default workspace;
2. Running my custom .gitpod.Dockerfile in a test:
   docker build -f .gitpod.Dockerfile -t gitpod-dockerfile-test .
3. Finally, I could see the error messages related to the problems of my Dockerfile.

I hope it helps someone.

[geropl]

Scheduled this, as the user experience feels really broken 😕 .

@felipealencar Thank you for sharing the workaround! This is indeed what's happening: The headless task failed: exit status 1 error is always emitted if there's some user-error with the image build.
That could be anything from error pulling the image, to a failing Docker command inside the docker file.

[ChamMach]

Yep, thanks for the workaround! I realized it too by simply building the problematic Dockerfile on another gitpod.

Thanks to the team for taking this issue into account too! 🙏

[jankeromnes]

Maybe this error is related? Error: no log stream for this image build (internal)

[csweichel]

@jankeromnes @meysholdt looks like a certificate issue with the new application clusters:

{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","buildID":"ca0da695-1b59-4827-9adf-f2054be7be7d","error":"cannot connect to supervisor:\n    github.com/gitpod-io/gitpod/image-builder/pkg/orchestrator.findTaskLogURL\n        github.com/gitpod-io/gitpod/image-builder/pkg/orchestrator/monitor.go:396\n  - Get \"https://ca0da695-1b59-4827-9adf-f2054be7be7d.ws.gitpod.io/_supervisor/v1/status/tasks\": GET https://ca0da695-1b59-4827-9adf-f2054be7be7d.ws.gitpod.io/_supervisor/v1/status/tasks giving up after 11 attempt(s): Get \"https://ca0da695-1b59-4827-9adf-f2054be7be7d.ws.gitpod.io/_supervisor/v1/status/tasks\": x509: certificate is valid for gitpod.io, not ca0da695-1b59-4827-9adf-f2054be7be7d.ws.gitpod.io","level":"warning","message":"headless log listener failed","serviceContext":{"service":"image-builder-mk3","version":"commit-40cfaec3f92cc6cadc42dfd5e6028b8c3049a16c"},"severity":"WARNING","time":"2022-05-17T22:13:32Z"}

[meysholdt]

In case the above error message is from production, then the request
Get \"https://ca0da695-1b59-4827-9adf-f2054be7be7d.ws.gitpod.io/_supervisor/v1/status/tasks
looks wrong because we don't have a ws-cluster named ws in prod.
Requesting the wrong URL would explain why the domains from the certs don't match.

There was a change in certs indeed (https://github.com/gitpod-io/ops/pull/2228), but I'm not sure it's related. CC @easyCZ .

[easyCZ]

If needed, even for a test, we can revert https://github.com/gitpod-io/ops/pull/2228 to eliminate it as the source.

[andrew-farries]

Working on the assumption that it's the workspace URL that's wrong and not the certificates, these are set here:

gitpod/install/installer/pkg/components/ws-manager/configmap.go

Lines 140 to 141 in faa6b30

	WorkspaceURLTemplate: fmt.Sprintf("https://{{ .Prefix }}.ws.%s", ctx.Config.Domain),
	WorkspacePortURLTemplate: fmt.Sprintf("https://{{ .WorkspacePort }}-{{ .Prefix }}.ws.%s", ctx.Config.Domain),

so if the hardcoded ws component in the string is incorrect, then it looks like we missed adding some configuration to the installer to allow this to vary.

But if the URL is wrong, why do we see an error about certs rather than have the request time out?

[andrew-farries]

This is how we used to construct these URLs in the helm chart:

gitpod/chart/templates/ws-manager-configmap.yaml

Lines 74 to 75 in 2f988c9

	"urlTemplate": "https://{{"{{ .Prefix }}"}}.ws{{- if $.Values.installation.shortname -}}-{{ $.Values.installation.shortname }}{{- end -}}.{{ $.Values.hostname }}",
	"portUrlTemplate": "https://{{"{{ .WorkspacePort }}"}}-{{"{{ .Prefix }}"}}.ws{{- if $.Values.installation.shortname -}}-{{ $.Values.installation.shortname }}{{- end -}}.{{ $.Values.hostname }}",

so it does look as though we need to set them via the installer in a similar way...

[mteam88]

Really struggling without this! @andrew-farries your pull request looks good!

[andrew-farries]

Re-opening this as #10127 had to be reverted because it broke preview environments (see internal discussion).

The installer (which we also use to deploy to preview environments) defaults to setting:

gitpod/install/installer/pkg/config/v1/config.go

Line 50 in b765e67

cfg.Metadata.InstallationShortname = "default" // TODO(gpl): we're tied to "default" here because that's what we put into static bridges in the past

Which breaks when used as part of the URL template strings in a preview env installation (we don't have a cert for that domain, and I'm not sure image build logs would be available there anyway).

Maybe @geropl can provide some context about why we need that default when he is back.

[WesleyMConner]

Can you add a little color on how best to debug?

Historically, I have used a three step process.

1. Performed a docker build of the prospective .gitpod.Dockerfile (in Gitpod itself).

---

docker build -f .gitpod.Dockerfile -t gitpod-dockerfile-test .

2. If "integrated chat" on status bar non functional #1 ran without issues, launch a local container (still in Gitpod) and do some spot checks.

---

docker run --rm -it gitpod-dockerfile-test /bin/bash

3. Commit the changes and fire up a new Gitpod instance (with the revised .gitpod.Dockerfile) and check for expected behavior.

Lately, I get through steps 2 and 3 with no obvious issues. Then, run into the "Error: headless task failed" message. Is there a step I can/should be doing and am missing OR is there something happening here that isn't easily smoked out?

Thanks,
Wes