ci(ssh): revert using ssh-compute action & increase sshd connection limit

[gustavovalverde]

Motivation

We've been having multiple errors (more than before) after implementing ssh-compute

Solution

• Revert the ssh-compute implementation
• Stop using IAP for TCP forwarding as there are known limitations which might be counterproductive for our use cases
• Keep using sudo in some docker commands as we might be connecting with a non-root user to the VMs
• Set the MaxStartups setting in sshd to 500 to avoid a failing SSH connection causes GitHub Action to fail

Note: tj-actions/changed-files file comparison was failing in this PR and it has been failing in the main branch, so we're adding a fix here too as this would halt the PR from merging. Here's the actual explanation on this fix tj-actions/changed-files#639 (comment)

Closes #5358
Closes #5365
Fixes #5362
Fixes #5361

Review

If CI passes, anyone can review this PR

Reviewer Checklist

• Will the PR name make sense to users?
  • Does it need extra CHANGELOG info? (new features, breaking changes, large changes)
• Are the PR labels correct?
• Does the code do what the ticket and PR says?
• How do you know it works? Does it have tests?
  • Tested I've tested this is working doing a manual SSH into the VM and confirming the file has changed with the actual configuration being used in the script file gcp-vm-startup-script.sh

Follow Up Work

We might also want to wait up to 90 seconds after a VM has been created, just so we're sure all configurations are complete.

[teor2345]

Looks great!

I'd like to check that the sshd connection limit adjustment actually worked before we merge this, since we're calling the script a different way now.

We might also want to use the bullseye image for the instances.

[gustavovalverde]

Sorry for adding chore: fix tj-actions/changed-files file comparison to this PR. But it's annoying having an ❌ in this PR and this will be pulled to main, fixing it here is minimal.

Edit: I had to fix it here anyways, as it wouldn't merge otherwise

[gustavovalverde]

CI is failing with a new (unrelated?) error 🥲

e2fsck: Cannot continue, aborting.
/dev/sdb is in use.

[teor2345]

CI is failing with a new (unrelated?) error 🥲

e2fsck: Cannot continue, aborting.
/dev/sdb is in use.

We've changed two things related to the disk in this PR:

• bullseye image, which worked in PR fix(ci): Increase the Google Cloud instance sshd connection limit #5365: https://github.com/ZcashFoundation/zebra/actions/runs/3216898611/jobs/5263285350#step:11:103
• moving the script to the image creation, which is new

Here are some things we could try:

• checking the server logs
• find out what is using the disk, by calling lsof when e2fsck or resize2fs fail
• moving e2fsck and resize2fs to the startup script

Is there anything else we could try?

[teor2345]

Is there anything else we could try?

• Using ssh to adjust the sshd config, like PR fix(ci): Increase the Google Cloud instance sshd connection limit #5365

That would only work if the instance startup script is causing the disk to be used.

[teor2345]

@gustavovalverde I just realised that these disk resize commands are only needed when we change the disk size.

Specifically, they are only needed between:

• when we merge the change to main, and
• when we stop using cached states with the smaller size (after the new larger full sync image is generated, or we've created new updated cached states with resized disks).

So let's ignore the failure for now, and fix it if it becomes a problem after #5085 ?

[gustavovalverde]

So let's ignore the failure for now, and fix it if it becomes a problem after #5085 ?

If a re-run fixes the issue, we can ignore it for now. I'm anyways testing a PR here to handle this repartitioning: #5371

[teor2345]

Thanks, looks good, let's get it fixed!