fix(ci): Increase the Google Cloud instance sshd connection limit

[teor2345]

Motivation

SSH connections to Google Cloud test instances are failing with:

ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4010: 'destination read failed'].
kex_exchange_identification: Connection closed by remote host

https://github.com/ZcashFoundation/zebra/actions/runs/3203311358/jobs/5234000346#step:6:106

Closes #5362

Reference

Weirdly, none actually try to authenticate to open a session.

Some spiders and services like Shodan scans public ipv4 addresses for open services, e.g. salt masters, ftp servers, RDPs, and also SSH services. These spiders usually only connect to the services without doing any valid authentication steps.

https://serverfault.com/questions/1015547/what-causes-ssh-error-kex-exchange-identification-connection-closed-by-remote/1015554#1015554

I got this error when using docker command with remote host
docker -H ssh://user@server compose up
after some digging i found on my remote server in auth logs (/var/log/auth.log) this:
Aug 8 14:51:46 user sshd[1341]: error: beginning MaxStartups throttling
Aug 8 14:51:46 user sshd[1341]: drop connection #10 from [some_ip]:32992 on [some_ip]:22 past MaxStartups

https://stackoverflow.com/questions/67000681/kex-exchange-identification-connection-closed-by-remote-host/73292521#73292521

MaxStartups
Specifies the maximum number of concurrent unauthenticated connections to the SSH daemon.
Additional connections will be dropped until authentication succeeds or the LoginGraceTime
expires for a connection.

https://man7.org/linux/man-pages/man5/sshd_config.5.html

Solution

Allow up to 500 unauthenticated connections to Google Cloud instances.
This tolerates a larger number of port scanning bots.

Review

These errors are blocking most other PR merges.

Reviewer Checklist

• Will the PR name make sense to users?
  • Does it need extra CHANGELOG info? (new features, breaking changes, large changes)
• Are the PR labels correct?
• Does the code do what the ticket and PR says?
• How do you know it works? Does it have tests?

Follow Up Work

If this change works, we should:

• revert the change to the ssh-compute action in PRs ci(ssh): connect using ssh-compute action by Google #5330, fix(ci): Show logs after each Google Cloud test job #5358, and fix(ci): Increase the Google Cloud instance sshd connection limit #5365, because they make test logging much worse
• modify the debian Docker image we use for Google Cloud instances, rather than editing the file after the instance has started using a SSH connection

[teor2345]

Finally got this working:

root 667 0.0 0.0 6324 4800 ? Ss 04:17 0:00 sshd: /usr/sbin/sshd -D -e [listener] 0 of 10-100 startups
..
root 667 0.0 0.0 6324 4680 ? Ss 04:17 0:00 sshd: /usr/sbin/sshd -D -e [listener] 0 of 500-500 startups

https://github.com/ZcashFoundation/zebra/actions/runs/3216832174/jobs/5259141110#step:9:507

[teor2345]

One test had a bug, but I think it is caused by the compute-ssh action changes in PR #5330:

client_loop: send disconnect: Broken pipe

https://github.com/ZcashFoundation/zebra/actions/runs/3216898611/jobs/5259351189#step:6:6089

Previously, this broken pipe error was handled using a tee option. We can't adjust the same option on ssh until we revert PR #5330. (And we might not even need to if we do the revert.)

[gustavovalverde]

Superseded by #5367

[teor2345]

Superseded by #5367

I think this PR fixes some of the ssh connection issues we were having before #5330, but I'll need to rewrite it to use the original ssh command syntax.

[teor2345]

Ah, I see you covered that in the new PR