Failed key preparations can result in extraneous keys being created in your KeyManager

[keeganwitt]

We had an AWS account in which SPIRE created 16,000 keys it wasn't using. If we had not caught this, we would have been charged $1 per key per month, so an additional $16,000 would have been charged to us. In this example we are using AWS KMS, but this can apply to other KeyManager plugins as well (such Azure or GCP).

The problem is that PrepareX509CA() and PrepareJWTKey() in manager.go have no logic to delete the newly created key if the function fails at some point after it's creation. For example, if it fails to publish the JWT or if getting the upstream server to sign the X.509 key fails.

[keeganwitt]

We do also have evidence of the server pod crashlooping (106 times). So, another question with this issue is how to make the key creation an atomic operation.