DLPX-76794 recovery_sync dropbear failure causes delphix-bootcount service to fail on Ubuntu 20.04

[pzakha]

The recovery_sync process copies the ssh keys from the main environment into the boot environment and converts them to a format that the dropbear ssh server understands, by using dropbearconvert. On Ubuntu 20.04 this process is failing with:

./usr/lib/dropbear/dropbearconvert openssh dropbear /etc/ssh/ssh_host_dsa_key etc/dropbear/dropbear_dsa_host_key
Error: Unrecognised key type
Error reading key from '/etc/ssh/ssh_host_dsa_key'

@pcd1193182 discovered that this was caused by all SSH keys now being generated with the proprietary OPENSSH header:

-----BEGIN OPENSSH PRIVATE KEY-----

instead of a header that is specific to the given key's type:

-----BEGIN RSA PRIVATE KEY-----
-----BEGIN EC PRIVATE KEY-----

By default ssh-keygen generates a key in the newer RFC4716 format, which was used on both Ubuntu 18.04 and Ubuntu 20.04, however on Ubuntu 20.04 the header has changed. The dropbearconvert utility on Ubuntu 20.04 doesn't know how to handle they keys with an OPENSSH header, however that was fixed in the Ubuntu 20.10 package (See mkj/dropbear#91). One workaround is to instead use a key that is generated in PEM format, which is always accepted by dropbearconvert.

Therefore there are 3 ways that this problem could be fixed:

1. Generate the keys in the PEM format from the beginning by modifying cloud-init.
2. Keep generating the keys in the better RFC4716 format but convert them to PEM before feeding them into dropbearconvert.
3. Update the dropbear package so that it works with that new header.

• Option 3 would have probably been the cleanest one, however we do not have an easy way to pull a package from a different archive, so it is the least practical one.

• Option 1: while it's unclear what are the downsides of using the older key format on the engine, forcing it to use an old format just to satisfy the dropbearconvert utility sounds like a change that could a bigger impact.

• Option 2, while introducing an extra step in the recovery_sync process, seems to have the least impact on the system while remaining simpler to implement that option 3. Hence I've chosen option 2.

Testing

• Checked that keys converted into PEM format are indeed the same as the default RFC4716 format using ssh-keygen -l.
• Manually tested that drobearconvert works using this method
• ab-pre-push on master: http://selfservice.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build-orchestrator-pre-push/5908/
• ab-pre-push on focal: http://selfservice.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build-orchestrator-pre-push/5909/ (verified that delphix-bootcount service doesn't fail any longer)