kola: support for ssh RSA key types other than RSA-SHA1

[dustymabe]

Basically we need to support a key type other than RSA-SHA1 so that kola can ssh in to F33+.

The short term plan is to only re-enable RSA-SHA1 briefly while we wait for an upstream feature [1] to be implemented. We had moved to an ecdsa key [2] but AWS doesn't support non RSA keys [3] so we reverted it for now in [4].

[1] golang/go#37278 (comment)
[2] #1749
[3] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#how-to-generate-your-own-key-and-import-it-to-aws
[4] #1767

Once the upstream golang issue is fixed let's pick up that work in mantle/kola.

[darkmuggle]

@dustymabe isn't this closed with #1749?

[dustymabe]

@dustymabe isn't this closed with #1749?

See #1767

[dustymabe]

Temporarily worked around this in the configs. This is a very short term change. We don't want to have f33 go to testing or stable with it in place.

[dustymabe]

Benjamin pointed out that the changes in Fedora and eventually openssh don't actually mean we have to change the key we're generating, but rather the key exchange algorithm. In order to do that we need to update our ssh client (kola use of the ssh library in this case) to use the new preferred algorithms. It seems we do still want the code referenced in my comment at golang/go#37278 (comment) so we can update kola to do that. So this issue still stands and we'll monitor to see if that code lands this week or next week. If not, we'll switch to an ecdsa key and just disable the afterburn ssh key test for now.

cc @bgilbert for fact checking.

[bgilbert]

Right. We could disable the Afterburn test exclusively on AWS. It's possible that we might still need to pass AWS a fake RSA key to make the API happy.

[dustymabe]

ok we're working around this temporarily by using an ecdsa key for Ignition/other cloud providers and using a throwaway key for AWS, and also disabling the fcos.ignition.v3.noop test on AWS. We should be able to revert the "workaround" commit and go back to using a single rsa key when we update the golang libraries.

[travier]

https://aws.amazon.com/about-aws/whats-new/2021/08/amazon-ec2-customers-ed25519-keys-authentication/

[bgilbert]

Excitingly, Azure doesn't support Ed25519.

[dustymabe]

Excitingly, Azure doesn't support Ed25519.

I wonder if there is an issue somewhere where we could follow updates on that.

Anywho, we should be able to just switch on type of key the way we currently do (see ac54a8a) and re-enable the fcos.ignition.misc.empty and fcos.ignition.v3.noop tests for AWS.

[bgilbert]

Anywho, we should be able to just switch on type of key the way we currently do (see ac54a8a) and re-enable the fcos.ignition.misc.empty and fcos.ignition.v3.noop tests for AWS.

To be clear, are you proposing generating different key types for different platforms? There's no non-RSA key type that's supported by every platform.

[dustymabe]

To be clear, are you proposing generating different key types for different platforms? There's no non-RSA key type that's supported by every platform.

we can just use EDCSA in most places (what we're doing now) and then switch AWS to Ed25519 instead of RSA, which is what we're currently doing:

coreos-assembler/mantle/platform/machine/aws/flight.go

Lines 61 to 78 in 602bd2a

	// We have worked around the golang library limitation for
	// keyexchange algorithm by switching to an ecdsa key in
	// network/ssh.go. However, AWS requires an rsa key. For now
	// (until we get an updated golang library) we'll just satisfy
	// the requirement by using a fake key and disabling the
	// fcos.ignition.misc.empty and fcos.ignition.v3.noop tests on AWS.
	// https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#how-to-generate-your-own-key-and-import-it-to-aws
	key, err := platform.GenerateFakeKey()
	if err != nil {
	af.Destroy()
	return nil, err
	}
	if err := api.AddKey(af.Name(), key); err != nil {
	af.Destroy()
	return nil, err
	}
	af.keyAdded = true

Maybe there's something I'm missing?

[bgilbert]

Again, just to be clear, your proposal is to plumb a platform-specific key type into the code that generates the real keys? The thing that's confusing me is that the implication that we're doing something similar now, which we are not. The snippet you quoted generates a non-operable key for AWS.

[dustymabe]

Indeed. As is often the case things are more complicated than they seem. I saw that we were generating an RSA key for AWS only and assumed we could just do the same thing but make it an ed25519 key instead. I think what was confusing to me is that this key isn't usable for reasons other than it being an RSA key and our crypto policies not allowing it because of the golang library unimplented feature.

I think you're implying to make it usable we'd need to update other parts of the code.

[bgilbert]

Right, we're generating an unused RSA key there only because the API requires that we provide something.

I do think we could plumb a key type argument into the key generation code; the call stack is NewFlight -> NewBaseFlight -> NewBaseFlightWithDialer -> NewSSHAgent. In principle we could be hacky and have NewBaseFlightWithDialer condition on the existing platform-name argument, but I think it's not too much more work to be explicit.

It doesn't look like there's been much movement on golang/go#37278.

[bgilbert]

It turns out that Azure doesn't support ECDSA either, so kola currently can't run tests on Azure (#2445).

[bgilbert]

golang/go#37278 was fixed upstream, but it appears to be scoped to SSH host keys and not pubkey authentication. I think we need golang/go#39885 for that.

[jmarrero]

@bgilbert does that mean that we are still blocked on this until golang/go#39885 is done?

[bgilbert]

Yup.