1427 - Add DNS and cert cleanup into the preview env cron

[liam-j-bennett]

Description

Added two functions to cleanup DNS records and certificates

Related Issue(s)

Fixes https://github.com/gitpod-io/ops/issues/1427

Release note

NONE

How to test

Create a new branch and then delete after all resources are created
Check the DNS and cert resources are correctly removed

[ArthurSens]

Would it make sense to implement DNS deletion using the same nodejs library we use to create them? 🙂

https://github.com/gitpod-io/gitpod/blob/main/.werft/util/gcloud.ts

[liam-j-bennett]

Would it make sense to implement DNS deletion using the same nodejs library we use to create them? 🙂

https://github.com/gitpod-io/gitpod/blob/main/.werft/util/gcloud.ts

Changed to use a util function

[ArthurSens]

Changed to use a util function

Nice, thanks Liam! 🙂

Another question: when implementing the creation, I had to add some checks to guarantee we don't try to recreate a DNS record that already exists. Do you know what would happen if we try to delete a DNS that does not exist?

[liam-j-bennett]

Changed to use a util function

Nice, thanks Liam! 🙂

Another question: when implementing the creation, I had to add some checks to guarantee we don't try to recreate a DNS record that already exists. Do you know what would happen if we try to delete a DNS that does not exist?

It returns a 404 (I hate delete APIs that do this). I'll add an ignore to the callback, as I don't think we would ever care if the delete returns this, as the record is gone either way.

[liam-j-bennett]

So earlier today, I tested this and it worked, with several branches completedly cleaned up. However, after some extra changes, I have not been able to test again, along with werft being unstable and therefore hard to test.

Since rebasing on main, it seems the branches are not being picked up:

1. New branch and preview env created https://werft.gitpod-dev.com/job/gitpod-build-lb-test-dns-cert-cleanup3.0
2. Branch deleted
3. Run job to clean up https://werft.gitpod-dev.com/job/gitpod-custom-lb-1427-add-dns-cert-cleanup.7/logs

The branch is in the preview envs log, but does not get cleaned up, even though the branch doesn't exist.

[mads-hartmann]

@ArthurSens this is ready for another review. This has since been rebased on main which included the changes from #9225 which it appears unfortunately had some issues when checking for DB activity in the preview environments where the Werft job would complete prematurely without trying to clean up preview environments - to debug this I had to improve the Werft logging as well as simply the logic for deciding if a preview environment is stale due to DB inactivity.

I apologise for the messy diff. I was able to verify it now deletes certificates and dns records in this job

[ArthurSens]

Looks pretty nice!

The only thing that worried me a bit was the number of slices 😅. If I understand honeycomb correctly, we pay for every new span, which is created for every new slice. So this amount of new slices sounds like a bit overkill. I'm approving anyways because when comparing to traces coming from production, I believe this is almost nothing 😛

Another thing I've noticed is that some of those slices' span is not being closed correctly 🤔

Also not a blocker for this PR, but I think it could be worth adding an issue for technical debt?

[mads-hartmann]

@ArthurSens Good catch with the span not being closed 🙏 I just pushed a commit that I believe will fix it. Given it's such a small change I don't think I have to manually test again - I'm happy to merge so it can run over night and I can check the output tomorrow.

[mads-hartmann]

@ArthurSens Just realised I didn't need to request re-review based on the new rules. Sorry about that ☺️