feat(jans-pycloudlib): add AWS secret manager support as a secret layer for holding CN configuration

[moabu]

Similar to the google secret handler, an AWS secret handler should be added to support multi regional setups with AWS secret manager has the holder of the secret and configuration output.

---

Things to consider:

• Support for AWS KMS key ID (by default using aws/secretsmanager) [NEED TEST ON REMOTE AWS]

  Reference:

  1. https://docs.aws.amazon.com/secretsmanager/latest/userguide/security-encryption.html
  2. https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/create-secret.html

• JSON structure [DONE]

  Excerpt from https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/create-secret.html:

  The text data to encrypt and store in this new version of the secret. We recommend you use a JSON structure of key/value pairs for your secret value.

• Compression support (lzma vs bz2) [DONE]

  Compress secrets value to reduce the size using lzma.

• SecretString type for configs (non-sensitive configuration) and SecretBinary for secrets (will produce base64 value) [DONE]

• Secrets naming [DONE]

  Excerpt from https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/create-secret.html

  The secret name can contain ASCII letters, numbers, and the following characters: /_+=.@-
  Do not end your secret name with a hyphen followed by six characters. If you do so, you risk confusion and unexpected results when searching for a secret by partial ARN. Secrets Manager automatically adds a hyphen and six random characters after the secret name at the end of the ARN.

• AWS credentials and region support [DONE]

• Replication of secrets to other regions [NEED TEST ON REMOTE AWS]

[iromli]

As we have configs and secrets layers, we can utilize different type of AWS secretsmanager value.

1. For configs (non-sensitive configuration), we can use SecretString type, for example:

   » awslocal secretsmanager create-secret --name hostname --secret-string demoexample.jans.io
   
   » awslocal secretsmanager get-secret-value --secret-id hostname
   {
       "ARN": "arn:aws:secretsmanager:us-east-1:000000000000:secret:hostname-PUrbYB",
       "Name": "hostname",
       "VersionId": "04084d42-730b-4412-b102-8fe577fd169f",
       "SecretString": "demoexample.jans.io",
       "VersionStages": [
           "AWSCURRENT"
       ],
       "CreatedDate": 1668801471.205548
   }
   
   

2. For secrets, we can use SecretBinary type, for example:

   » awslocal secretsmanager create-secret --name ssl_cert --secret-binary this-is-supposed-to-be-a-secret
   
   » awslocal secretsmanager get-secret-value --secret-id ssl_cert
   {
       "ARN": "arn:aws:secretsmanager:us-east-1:000000000000:secret:ssl_cert-gxSsyG",
       "Name": "ssl_cert",
       "VersionId": "398d9038-ad05-498a-9164-4e7142faad4b",
       "SecretBinary": "dGhpcy1pcy1zdXBwb3NlZC10by1iZS1hLXNlY3JldA==",
       "VersionStages": [
           "AWSCURRENT"
       ],
       "CreatedDate": 1668801851.753725
   }
   
   

   Which can be decoded (if necessary), as seen below:

   » awslocal secretsmanager get-secret-value --secret-id ssl_cert | yq '.SecretBinary' | base64 -d
   this-is-supposed-to-be-a-secret
   

[moabu]

Agreed. However, I think we need to unify the method . With google secrets we are also encrypting and compressing the data that is pushed to the secret manager. I think we need to either remove that or also enforce the same with AWS

[iromli]

Lets keep the Google secrets as is to avoid regression. For AWS secrets, we may add compression before saving it as SecretBinary type.

[iromli]

Though in my opinion, adding compression likely will make it harder to check the value from official AWS CLI/client apps as those need to decode and/or decompress the value first. Not as straight as we used to have when dealing with native k8s or Vault secrets.

[moabu]

Right , though the compression came out of the need to lower the size of the payload since upstream clouds have limits on the size of each secret. Since we have full files in our secrets that becomes an issue so we have to compress to make sure we keep the payload and secret low in size and add a how to for dealing with checking the secrets data.