[prebuilds] no prebuilds for inactive repos

[svenefftinge]

Description

This change prevents prebuilds for repositories that have been inactive (no regular workspace starts) for at least a week.
This would have prevented 98835 unnecessary prebuilds in the last 11 days (vs. 28174 prebuilds on repos without projects that are still active).

Related Issue(s)

See also my comment here #9898 (comment)
Fixes #9962

How to test

(AT) Unfortunately this isn't easy to test, because this requires using the old-style /prebuild# triggers and DB connection to verify prebuild statuses. Here is the SQL output of my test run, injecting the following markers:

• (t1) manually shifted creation time to the past, after adding "inactivityPeriodForRepos": 7 to server-config
• (t2) realized that I need to to use /prebuild# trigger instead of having a project
• (t3) verified that prebuilds are cancelled because there were no recent workspaces found for this repo
• (t4) then started a new workspace for the repo to show activity
• (t5) verified that prebuilds are now executed because recent activity was registered

mysql> select id, cloneURL, creationTime, type from d_b_workspace order by creationTime desc;
+----------------------------+---------------------------------------+--------------------------+----------+
| id                         | cloneURL                              | creationTime             | type     |
+----------------------------+---------------------------------------+--------------------------+----------+
(t5)
| alex736-foobar-i98zt3zqdfd | https://gitlab.com/alex736/foobar.git | 2022-05-19T07:26:49.600Z | prebuild |
(t4)
| alex736-foobar-3s469dy0oqp | https://gitlab.com/alex736/foobar.git | 2022-05-19T07:25:13.766Z | regular  |
(t3)
| alex736-foobar-fo239hhoqpj | https://gitlab.com/alex736/foobar.git | 2022-05-19T07:06:29.873Z | prebuild |
| alex736-foobar-mf28ya5g7en | https://gitlab.com/alex736/foobar.git | 2022-05-19T07:05:07.350Z | prebuild |
| alex736-foobar-yy5cq1z62sl | https://gitlab.com/alex736/foobar.git | 2022-05-19T06:59:56.618Z | prebuild |
| alex736-foobar-rfw34kst9kn | https://gitlab.com/alex736/foobar.git | 2022-05-19T06:56:26.167Z | prebuild |
| alex736-foobar-lvr6iyg35ev | https://gitlab.com/alex736/foobar.git | 2022-05-19T06:40:23.744Z | prebuild |
| alex736-foobar-ki4qzvqcmhz | https://gitlab.com/alex736/foobar.git | 2022-05-19T06:37:12.455Z | prebuild |
| alex736-foobar-v6o5cv1wdop | https://gitlab.com/alex736/foobar.git | 2022-05-19T06:34:30.626Z | prebuild |
| alex736-foobar-acczbzbds29 | https://gitlab.com/alex736/foobar.git | 2022-05-19T06:33:46.042Z | prebuild |
(t1)
| alex736-foobar-ddut8jxkxx8 | https://gitlab.com/alex736/foobar.git | 2022-05-10T06:29:18.215Z | regular  |
| alex736-foobar-ikpc6tcfosd | https://gitlab.com/alex736/foobar.git | 2022-05-10T06:29:18.215Z | regular  |
+----------------------------+---------------------------------------+--------------------------+----------+
12 rows in set (0.00 sec)

mysql> select state, buildWorkspaceId, creationTime from d_b_prebuilt_workspace order by creationTime desc;
+-----------+----------------------------+----------------------------+
| state     | buildWorkspaceId           | creationTime               |
+-----------+----------------------------+----------------------------+
(t5)
| building  | alex736-foobar-i98zt3zqdfd | 2022-05-19 07:26:49.026490 |
(t3)
| aborted   | alex736-foobar-fo239hhoqpj | 2022-05-19 07:06:29.254310 |
| aborted   | alex736-foobar-mf28ya5g7en | 2022-05-19 07:05:06.784755 |
(t2)
| available | alex736-foobar-yy5cq1z62sl | 2022-05-19 06:59:56.086979 |
| available | alex736-foobar-rfw34kst9kn | 2022-05-19 06:56:25.561247 |
| available | alex736-foobar-lvr6iyg35ev | 2022-05-19 06:40:23.782301 |
| available | alex736-foobar-ki4qzvqcmhz | 2022-05-19 06:37:12.486431 |
| available | alex736-foobar-v6o5cv1wdop | 2022-05-19 06:34:30.652669 |
| available | alex736-foobar-acczbzbds29 | 2022-05-19 06:33:46.069676 |
+-----------+----------------------------+----------------------------+
9 rows in set (0.00 sec)

Release Notes

Avoid prebuilding repositories were no workspaces got started for recently.

[svenefftinge]

/werft run

👍 started the job as gitpod-build-se-no-prebuilds-inactive.3
(with .werft/ from main)

[svenefftinge]

/werft run with-clean-slate-deployment=true

👍 started the job as gitpod-build-se-no-prebuilds-inactive.9
(with .werft/ from main)

[geropl]

https://github.com/gitpod-io/gitpod/pull/9976/files#r872044707

No longer using the GENERATED COLUMN, thus releasing the break.

[AlexTugarev]

I've added ce6b918 to decouple deployment from enablement. That means, we can migrate as much entries as needed and then reconfigure server to enable this feature.

[jankeromnes]

Thanks for the review request! And for adding the config flag.

I'm a bit worried now, because this Pull Request is growing more and more in complexity. At first it was a quick fix to disable prebuilds for inactive repo URLs, but now we have:

• some additional logic with a 255 character hash index (how well was this tested? has anyone calculated the collision risk?)
• a migration on d_b_workspace (that's likely going to take a long time and a lot of DB CPU, right?)
• a configuration change (which can lead to more bugs / ops incidents / headaches)

This is starting to look a bit less like a skateboard. 😕

How about, instead, we purely and simply disable all prebuilds that do not have a project? That's something we want to do anyway soon, and it resolves the same problem in a very simple, risk-free way.

[svenefftinge]

some additional logic with a 255 character hash index (how well was this tested? has anyone calculated the collision risk?)

It doesn't add complexity as it is well encapsulated in a single function and the max length we had in the past 3 months is 159.

a migration on d_b_workspace (that's likely going to take a long time and a lot of DB CPU, right?)

Why is that migration problematic or taking a long time? It's adding an empty column.

a configuration change (which can lead to more bugs / ops incidents / headaches)

I agree that is a bit over the top as this was intended an intermediate fix, but really no reason to not move this forward.

How about, instead, we purely and simply disable all prebuilds that do not have a project? That's something we want to do anyway soon, and it resolves the same problem in a very simple, risk-free way.

There are way too many important active projects to "simply" do this without notice.

Can we please get this in now and move on?

[svenefftinge]

I've added ce6b918 to decouple deployment from enablement. That means, we can migrate as much entries as needed and then reconfigure server to enable this feature.

There is no need to be flexible about the activity time.
We could instead update the last 7 days of regular workspaces within the migration.

[jankeromnes]

Okay, giving this a closer look and a vigorous shake (i.e. testing it).

[jankeromnes]

Code is a bit more complicated than I'd like but still looks correct. Also, I tested this a lot, and it still works as advertised.

It's nice to be able to disable this by setting inactivityPeriodForRepos to 0 in the server configmap (also tested this, works).

So, approving, but I'm very much looking forward to us simply disabling prebuilds for non-projects (and deleting the /#prebuild/ manual prefix and all of its problems).

Thanks again! Ad astra 🚀

[jankeromnes]

I guess this can never be undefined right?

Suggested change

	cloneUrl?: string;
	cloneUrl: string;

[jankeromnes]

Micro-nit:

Suggested change

	* Copyright (c) 2021 Gitpod GmbH. All rights reserved.
	* Copyright (c) 2022 Gitpod GmbH. All rights reserved.