[prebuild] Not able to consistently view logs and start related workspaces

[kylos101]

Bug description

This happens intermittently to some users. When opening a prebuild to view its logs, they sometimes see the following, the prebuild (which normally takes 10m) times out after an hour, and then the workspace cannot be started:

Refreshing the page results in the same screen (it does not reload and then show the editor).

As a workaround, the prebuild can be triggered manually by either:

1. Building a URL to browse to it gitpod.io/#prebuild/https://github.com/your-org/your-repo
2. Pushing a new commit

Sometimes the logs are visible, but the workspace cannot be started, and they see an error ERR_SOCKET_TIMEOUT.

Steps to reproduce

This appears to be related when a prebuild is doing heavy network traffic, generally with pulling images via docker compose, or npm install.

Workspace affected

n/a

Expected behavior

The prebuild logs must be visible when a prebuild is running.

When a prebuild stops within a workspace, supervisor must reliably terminate the pod, rather than wait for the 1 hour to finish.

Example repository

n/a

Anything else?

While observing this behavior today, we saw a customer workload that normally takes 10 minute enter a state where supervisor became unhealthy after 10 minutes (so roughly when the prebuild would have finished if it were successful).

[axonasif]

@jmls you can subscribe here for updates 🙏

[kylos101]

@jldec hey there, this could be a bug with supervisor. Looping in @akosyakov and the IDE team, adding to their inbox.

[akosyakov]

Did someone from @gitpod-io/engineering-webapp already investigated whether we don't lose them somewhere between supervisor and dashboard?

[svenefftinge]

Really just a theory, but we are currently (over) packing nodes with prebuilds. So that they often run much slower than usual/locally, because they only get 0.7 CPUs (AFAIR).

So my theory is that in cases where a network request is initiated, but then due to CPU scheduling no further activity happens on that thread for a certain time, timeouts could be hit. We have seen lots of reports about very different prebuild execution times.

We should probably not schedule more prebuilds than we have CPU cores, because prebuilds are often CPU bound and in that sense have a very different usage profile than interactive workspaces.

[svenefftinge]

We should probably not schedule more prebuilds than we have CPU cores, because prebuilds are often CPU bound and in that sense have a very different usage profile than interactive workspaces.

@kylos101 independent of whether this would improve the situation described in this ticket, I believe reducing the density to a point where still 80% of CPU is used most of the time makes sense. cc @csweichel

[akosyakov]

It seems that it was npm network hiccup in user task. Increasing timeout seem to resolve it: https://gitpod.slack.com/archives/C01KGM9BH54/p1645714873540489?thread_ts=1645712044.304869&cid=C01KGM9BH54

[geropl]

I'm closing this issue because the network timeout is not an inherent problem by itself, and can be mitigated by increasing npm network timeouts: yarn install --network-timeout 60000

Nonetheless, we should have follow up issues for:

• Workspace: due to high CPU density prebuilds can start/execute very slowly (might depend on Rate limit prebuilds #5176 first to limit cost impact) @kylos101
• WebApp: improve user feedback on prebuilds startup/execution (allocation resources, pulling docker images, etc.) ("Prebuild in Progress" is often misleading or misbehaves. #8446)
• WebApp: rate-limiter for prebuilds (Rate limit prebuilds #5176)

/cc @JanKoehnlein