feat(ssh): enable OS Login for GCP test instances

[gustavovalverde]

Motivation

We've been dealing with several different issues related to SSH connections. So far one of the solutions was to have a fixed SSH-key to connect from GitHub Actions, but this also reduces flexibility and maintainability, and goes against our security-wise decision of not having long-lived artifacts (JSON tokens, SSH keys) without expiration, which could be used in any point in time.

This PR implements OS Login as workaround (and possible root cause solution)

Note: Not all changes to make this work are being done in GitHub Actions, as some tasks are tied to changes in the infrastructure

Fixes: #5494
Fixes: #5069

Specifications

https://cloud.google.com/compute/docs/oslogin#benefits_of_os_login

Designs

• Activate OS Login on specific VMs
• Give the GitHub Actions service account the required permissions to connect as a user

Solution

• Enable OS Login
• Expire OS Login SSH keys after 30 seconds (as we don't need those anymore)
• Delete any remaining keys (if any) at the end of the workflow run
• Use busybox instead of failing debian-11 container image ci: compute engine VMs have not been using the expected OS (debian-11) #5618
• Add sudo to a missing docker command

Review

Anyone from DevOps after running this PR successfully ~5+ times

Reviewer Checklist

• Will the PR name make sense to users?
  • Does it need extra CHANGELOG info? (new features, breaking changes, large changes)
• Are the PR labels correct?
• Does the code do what the ticket and PR says?
• How do you know it works? Does it have tests?

Follow Up Work

Configure OS-login project-wide after this PR has been merged.

[teor2345]

Sounds interesting, hope it works!

We've been dealing with several different issues related to SSH connections. ...
This PR implements OS Login as workaround (and possible root cause solution)

How will we test that this works before merging?
Do we need to run this PR multiple times to make sure SSH is more reliable?

Note: Not all changes to make this work are being done in GitHub Actions, as some tasks are tied to changes in the infrastructure

Can you document the infrastructure changes somewhere?

[gustavovalverde]

How will we test that this works before merging?
Do we need to run this PR multiple times to make sure SSH is more reliable?

Yes, running this PR multiple times is only thing I can think we have available at our disposal right now.

Can you document the infrastructure changes somewhere?

I'll be translating the manual changes to a terraform config

[gustavovalverde]

Seems like the authentication issue I've been having is related to google-github-actions/setup-gcloud#586

Which was fixed 12 hours ago.

[teor2345]

I've just realized that other running workflows will affect this implementation, as those will be continuously creating SSH key pairs and uploading the public key to GCP's metadata; and then failing with the following error message: Permission denied (publickey), as those keys can conflict with the ones being used by this implementation.

That's really unfortunate, it makes it tricky to test and merge.

How can we make sure that SSH is more reliable with this change?

Did you want to delay this change to the start of next week, so you and @dconnolly can fix any issues quickly?
(I won't be available, unfortunately.)

[gustavovalverde]

I started testing this with os-login enabled project wide.

Attempt #1: ✅ Failed, but wasn't an SSH error (https://github.com/ZcashFoundation/zebra/actions/runs/3440425995/jobs/5754296381#step:6:65)

Attempt #1.5: ✅ Success

[gustavovalverde]

I removed several SSH keys with the following command

for i in $(gcloud compute os-login ssh-keys list --format="table[no-heading](value.fingerprint)") --impersonate-service-account=github-service-account@zealous-zebra.iam.gserviceaccount.com; do 
  echo "$i";
gcloud compute os-login ssh-keys remove --key "$i" --impersonate-service-account=github-service-account@zealous-zebra.iam.gserviceaccount.com || true;
  done

And re-running. I'll update this command later once this run finishes.

I might also increase --ssh-key-expire-after to 5m as there's an edge case which can cause specific jobs to fail https://cloud.google.com/compute/docs/troubleshooting/troubleshooting-ssh-errors#:~:text=Your%20key%20expired%20and%20Compute%20Engine%20deleted%20your%20~/.ssh/authorized_keys%20file

[dconnolly]

I removed several SSH keys with the following command

for i in $(gcloud compute os-login ssh-keys list --format="table[no-heading](value.fingerprint)") --impersonate-service-account=github-service-account@zealous-zebra.iam.gserviceaccount.com; do 
  echo "$i";
gcloud compute os-login ssh-keys remove --key "$i" --impersonate-service-account=github-service-account@zealous-zebra.iam.gserviceaccount.com || true;
  done

And re-running. I'll update this command later once this run finishes.

I might also increase --ssh-key-expire-after to 5m as there's an edge case which can cause specific jobs to fail https://cloud.google.com/compute/docs/troubleshooting/troubleshooting-ssh-errors#:~:text=Your%20key%20expired%20and%20Compute%20Engine%20deleted%20your%20~/.ssh/authorized_keys%20file

Once added, should this PR move out of draft and be priortized to merge? We're seeing the 32KB limit being hit on these jobs in multiple places as the switch got flipped on the gcloud side

[gustavovalverde]

We're seeing the 32KB limit being hit on these jobs in multiple places as the switch got flipped on the gcloud side

Now that you say this, maybe those other jobs are the ones generating all the SSH-key "garbage" in GCP, even though I applied the policy on GCP side.

I think we can merge this, delete all keys and update all branches. Worst case scenario is reverting this, but it's hard to test with other jobs running and colliding with this. I'll move it out of draft and let you approve if you think this is a sane approach.

Let's wait for Deirdre

[arya2]

maybe those other jobs are the ones generating all the SSH-key "garbage" in GCP

This was my first thought when seeing it happen in the one other PR that has the Delete temporal SSH keys job

[gustavovalverde]

@Mergifyio refresh

[mergify]

refresh

✅ Pull request refreshed