Add support for AWS KMS as a secrets backend

README.md

@@ -52,18 +52,21 @@ For CI/CD usage, check out [ci/](https://github.com/deepmind/kapitan/tree/master
 Kapitan needs Python 3.6.
 
 **Install Python 3.6:**
-<br>Linux: `sudo apt-get update && sudo apt-get install -y python3.6-dev python3-pip python3-yaml`
-<br>Mac: `brew install python3 libyaml`
+
+ * Linux: `sudo apt-get update && sudo apt-get install -y python3.6-dev python3-pip python3-yaml`
+ * Mac: `brew install python3 libyaml`
 
 **Install Kapitan:**
 
 User (`$HOME/.local/lib/python3.6/bin` on Linux or `$HOME/Library/Python/3.6/bin` on macOS):
-```
+
+```shell
 pip3 install --user --upgrade kapitan
 ```
 
 System-wide (not recommended):
-```
+
+```shell
 sudo pip3 install --upgrade kapitan
 ```
 
@@ -88,7 +91,7 @@ These targets generate the following resources:
 
 ![demo](https://raw.githubusercontent.com/deepmind/kapitan/master/docs/demo.gif)
 
-```
+```shell
 $ cd examples/kubernetes
 
 $ kapitan compile
@@ -151,7 +154,6 @@ parameters:
         java_opts: ${elasticsearch:java_opts}
         replicas: ${elasticsearch:replicas}
         masters: ${elasticsearch:masters}
-...
 ```
 
 Or in the `mysql` class example, we declare the generic variables that will be shared by all targets that import the component and what to compile.
@@ -170,6 +172,7 @@ parameters:
         # If 'secrets/mysql/root/password' doesn't exist, it will gen a random b64-encoded password
         password: ?{gpg:mysql/root/password|randomstr|base64}
         # password: ?{gkms:mysql/root/password|randomstr|base64}
+        # password: ?{awskms:mysql/root/password|randomstr|base64}
 
   kapitan:
     compile:
@@ -319,7 +322,8 @@ optional arguments:
 ```
 
 These parameters can also be defined in a local `.kapitan` file, for example:
-```
+
+```shell
 $ cat .kapitan
 
 compile:
@@ -328,14 +332,15 @@ compile:
 ```
 
 This is equivalent to running:
-```
+
+```shell
 kapitan compile --indent 4 --parallelism 8
 ```
 
 To enforce the kapitan version used for compilation (for consistency and safety), you can add `version` to `.kapitan`:
-```
-$ cat .kapitan
 
+```shell
+$ cat .kapitan
 ...
 version: 0.19.0
 ```
@@ -375,7 +380,7 @@ java_opts for elasticsearch data role are: {{ inventory.parameters.elasticsearch
 
 #### Jinja2 jsonnet templating
 
-Such as reading the inventory within jsonnet, Kapitan also provides a function to render a Jinja2 template file. Again, importing "kapitan.jsonnet" is needed.
+Such as reading the inventory within jsonnet, Kapitan also provides a function to render a Jinja2 template file. Again, importing `kapitan.jsonnet` is needed.
 
 The jsonnet snippet renders the jinja2 template in templates/got.j2:
 
@@ -402,7 +407,7 @@ b64decode - base64 decode text e.g. {{ text | b64decode }}
 
 ### kapitan secrets
 
-Manages your secrets with GPG or Google Cloud KMS (beta), with plans to also support AWS KMS and Vault.
+Manages your secrets with GPG, Google Cloud KMS (beta) or AWS KMS (beta), with plans to also support Vault.
 
 The usual flow of creating and using an encrypted secret with kapitan is:
 
@@ -414,42 +419,45 @@ The usual flow of creating and using an encrypted secret with kapitan is:
     ```
     GPG: kapitan secrets --write gpg:mysql/root/password -t minikube-mysql -f <password file>
     gKMS: kapitan secrets --write gkms:mysql/root/password -t minikube-mysql -f <password file>
+    awsKMS: kapitan secrets --write awskms:mysql/root/password -t minikube-mysql -f <password file>
     OR use stdin:
-    echo -n '<password>' | kapitan secrets --write [gpg/gkms]:mysql/root/password -t minikube-mysql -f -
+    echo -n '<password>' | kapitan secrets --write [gpg/gkms/awskms]:mysql/root/password -t minikube-mysql -f -
     ```
     This will inherit the secrets configuration from minikube-mysql target, encrypt and save your password into `secrets/mysql/root/password`, see `examples/kubernetes`.
 
   - Automatically:<br>
     See [mysql.yml class](https://github.com/deepmind/kapitan/tree/master/examples/kubernetes/inventory/classes/component/mysql.yml). When referencing your secret, you can use the following functions to automatically generate, encrypt and save your secret:
     ```
-    randomstr - Generates a random string. You can optionally pass the length you want i.e. randomstr:32
-    rsa - Generates an RSA 4096 private key (PKCS#8). You can optionally pass the key size i.e. rsa:2048
-    rsapublic - Derives an RSA public key from a private key. Required argument is the private key file i.e. rsapublic:path/to/encrypted_private_key
-    base64 - base64 encodes your secret; to be used as a secondary function i.e. randomstr|base64
-    sha256 - sha256 hashes your secret; to be used as a secondary function i.e. randomstr|sha256. You can optionally pass a salt i.e randomstr|sha256:salt -> becomes sha256("salt:<generated random string>")
+    randomstr - Generates a random string. You can optionally pass the length you want i.e. `randomstr:32`
+    rsa - Generates an RSA 4096 private key (PKCS#8). You can optionally pass the key size i.e. `rsa:2048`
+    rsapublic - Derives an RSA public key from a private key. Required argument is the private key file i.e. `rsapublic:path/to/encrypted_private_key`
+    base64 - base64 encodes your secret; to be used as a secondary function i.e. `randomstr|base64`
+    sha256 - sha256 hashes your secret; to be used as a secondary function i.e. `randomstr|sha256`. You can optionally pass a salt i.e `randomstr|sha256:salt` -> becomes `sha256("salt:<generated random string>")`
     ```
 
 - Use your secret in your classes/targets, like in the [mysql.yml class](https://github.com/deepmind/kapitan/tree/master/examples/kubernetes/inventory/classes/component/mysql.yml):
-```
-users:
-  root:
-    # If 'secrets/mysql/root/password' doesn't exist, it will gen a random b64-encoded password
-    password: ?{gpg:mysql/root/password|randomstr|base64}
-```
+  ```
+  users:
+    root:
+      # If 'secrets/mysql/root/password' doesn't exist, it will gen a random b64-encoded password
+      password: ?{gpg:mysql/root/password|randomstr|base64}
+  ```
 
 - After `kapitan compile`, this will compile to the [mysql_secret.yml k8s secret](https://github.com/deepmind/kapitan/tree/master/examples/kubernetes/compiled/minikube-mysql/manifests/mysql_secret.yml). If you are part of the GPG recipients, you can see the secret by running:
-```
-kapitan secrets --reveal -f examples/kubernetes/compiled/minikube-mysql/manifests/mysql_secret.yml
-```
+  ```
+  kapitan secrets --reveal -f examples/kubernetes/compiled/minikube-mysql/manifests/mysql_secret.yml
+  ```
 
 To setup GPG for the kubernetes examples you can run:
-```
+
+```shell
 gpg --import examples/kubernetes/secrets/example\@kapitan.dev.pub
 gpg --import examples/kubernetes/secrets/example\@kapitan.dev.key
 ```
 
 And to trust the GPG example key:
-```
+
+```shell
 gpg --edit-key example@kapitan.dev
 gpg> trust
 Please decide how far you trust this user to correctly verify other users' keys
@@ -469,7 +477,7 @@ gpg> quit
 
 Rendering the inventory for the `minikube-es` target:
 
-```
+```shell
 $ kapitan inventory -t minikube-es
 ...
 classes:
@@ -609,7 +617,7 @@ With Kapitan, we worked to de-compose several problems that most of the other so
 
 3) ***Hierarchical inventory***: This is the feature that sets us apart from other solutions. We use the inventory (based on [reclass](https://github.com/salt-formulas/reclass)) to define variables and properties that can be reused across different projects/deployments. This allows us to limit repetition, but also to define a nicer interface with developers (or CI tools) which will only need to understand YAML to operate changes.
 
-4) ***Secrets***: We manage most of our secrets with kapitan using the GPG and Google Cloud KMS integrations. Keys can be setup per class, per target or shared so you can easily and flexibly manage access per environment. They can also be dynamically generated on compilation, if you don't feel like generating random passwords or RSA private keys, and they can be referenced in the inventory like any other variables. We have plans to support other providers such as AWS KMS and Vault, in addition to GPG and Google Cloud KMS.
+4) ***Secrets***: We manage most of our secrets with kapitan using the GPG, Google Cloud KMS and AWS KMS integrations. Keys can be setup per class, per target or shared so you can easily and flexibly manage access per environment. They can also be dynamically generated on compilation, if you don't feel like generating random passwords or RSA private keys, and they can be referenced in the inventory like any other variables. We have plans to support other providers such as Vault, in addition to GPG, Google Cloud KMS and AWS KMS.
 
 5) ***Canned scripts***: We treat scripts as text templates, so that we can craft pre-canned scripts for the specific target we are working on. This can be used for instance to define scripts that setup clusters, contexts or allow to run kubectl with all the correct settings. Most other solutions require you to define contexts and call kubectl with the correct settings. We take care of that for you. Less ambiguity, less mistakes.
 

examples/kubernetes/inventory/classes/common.yml

@@ -13,3 +13,5 @@ parameters:
             fingerprint: D9234C61F58BEB3ED8552A57E28DC07A3CBFAE7C
       gkms:
         key: 'projects/<project>/locations/<location>/keyRings/<keyRing>/cryptoKeys/<key>'
+      awskms:
+        key: 'alias/nameOfKey'

examples/kubernetes/inventory/classes/component/mysql.yml

@@ -24,3 +24,4 @@ parameters:
         # If 'secrets/mysql/root/password' doesn't exist, it will gen a random b64-encoded password
         password: ?{gpg:mysql/root/password|randomstr|base64}
         # password: ?{gkms:mysql/root/password|randomstr|base64}
+        # password: ?{awskms:mysql/root/password|randomstr|base64}

kapitan/cached.py

@@ -20,4 +20,5 @@
 inv_cache = {}
 gpg_obj = None
 gkms_obj = None
+awskms_obj = None
 dot_kapitan = {}

kapitan/cli.py

@@ -35,6 +35,7 @@
 from kapitan.refs.secrets.gpg import GPGSecret
 from kapitan.refs.secrets.gpg import lookup_fingerprints
 from kapitan.refs.secrets.gkms import GoogleKMSSecret
+from kapitan.refs.secrets.awskms import AWSKMSSecret
 
 from kapitan.errors import KapitanError
 
@@ -341,12 +342,28 @@ def secret_write(args, ref_controller):
         secret_obj = GoogleKMSSecret(data, key, encode_base64=args.base64)
         tag = '?{{gkms:{}}}'.format(token_path)
         ref_controller[tag] = secret_obj
+
+    elif token_name.startswith("awskms:"):
+        type_name, token_path = token_name.split(":")
+        key = args.key
+        if args.target_name:
+            inv = inventory_reclass(args.inventory_path)
+            kap_inv_params = inv['nodes'][args.target_name]['parameters']['kapitan']
+            if 'secrets' not in kap_inv_params:
+                raise KapitanError("parameters.kapitan.secrets not defined in {}".format(args.target_name))
+
+            key = kap_inv_params['secrets']['awskms']['key']
+        if not key:
+            raise KapitanError("No KMS key specified. Use --key or specify it in parameters.kapitan.secrets.awskms.key and use --target")
+        secret_obj = AWSKMSSecret(data, key, encode_base64=args.base64)
+        tag = '?{{awskms:{}}}'.format(token_path)
+        ref_controller[tag] = secret_obj
     else:
-        fatal_error("Invalid token: {name}. Try using gpg/gkms:{name}".format(name=token_name))
+        fatal_error("Invalid token: {name}. Try using gpg/gkms/awskms:{name}".format(name=token_name))
 
 
 def secret_update(args, ref_controller):
-    "Update secret gpg recipients/gkms key"
+    "Update secret gpg recipients/gkms/awskms key"
     # TODO --update *might* mean something else for other types
     token_name = args.update
     if token_name.startswith("gpg:"):
@@ -391,8 +408,25 @@ def secret_update(args, ref_controller):
         secret_obj.update_key(key)
         ref_controller[tag] = secret_obj
 
+    elif token_name.startswith("awskms:"):
+        key = args.key
+        if args.target_name:
+            inv = inventory_reclass(args.inventory_path)
+            kap_inv_params = inv['nodes'][args.target_name]['parameters']['kapitan']
+            if 'secrets' not in kap_inv_params:
+                raise KapitanError("parameters.kapitan.secrets not defined in {}".format(args.target_name))
+
+            key = kap_inv_params['secrets']['awskms']['key']
+        if not key:
+            raise KapitanError("No KMS key specified. Use --key or specify it in parameters.kapitan.secrets.awskms.key and use --target")
+        type_name, token_path = token_name.split(":")
+        tag = '?{{awskms:{}}}'.format(token_path)
+        secret_obj = ref_controller[tag]
+        secret_obj.update_key(key)
+        ref_controller[tag] = secret_obj
+
     else:
-        fatal_error("Invalid token: {name}. Try using gpg/gkms:{name}".format(name=token_name))
+        fatal_error("Invalid token: {name}. Try using gpg/gkms/awskms:{name}".format(name=token_name))
 
 
 def secret_reveal(args, ref_controller):
@@ -412,7 +446,7 @@ def secret_reveal(args, ref_controller):
 
 def secret_update_validate(args, ref_controller):
     "Validate and/or update target secrets"
-    # update gpg recipients/gkms key for all secrets in secrets_path
+    # update gpg recipients/gkms/awskms key for all secrets in secrets_path
     # use --secrets-path to set scanning path
     inv = inventory_reclass(args.inventory_path)
     targets = set(inv['nodes'].keys())
@@ -437,9 +471,14 @@ def secret_update_validate(args, ref_controller):
             recipients = None
 
         try:
-            key = kap_inv_params['secrets']['gkms']['key']
+            gkey = kap_inv_params['secrets']['gkms']['key']
+        except KeyError:
+            gkey = None
+
+        try:
+            awskey = kap_inv_params['secrets']['awskms']['key']
         except KeyError:
-            key = None
+            awskey = None
 
         for token_path in token_paths:
             if token_path.startswith("?{gpg:"):
@@ -465,16 +504,29 @@ def secret_update_validate(args, ref_controller):
                         ref_controller[token_path] = secret_obj
 
             elif token_path.startswith("?{gkms:"):
-                if not key:
+                if not gkey:
                     logger.debug("secret_update_validate: target: %s has no inventory gkms key, skipping %s", target_name, token_path)
                     continue
                 secret_obj = ref_controller[token_path]
-                if key != secret_obj.key:
+                if gkey != secret_obj.key:
+                    if args.validate_targets:
+                        logger.info("%s key mismatch", token_path)
+                        ret_code = 1
+                    else:
+                        secret_obj.update_key(gkey)
+                        ref_controller[token_path] = secret_obj
+
+            elif token_path.startswith("?{awskms:"):
+                if not awskey:
+                    logger.debug("secret_update_validate: target: %s has no inventory awskms key, skipping %s", target_name, token_path)
+                    continue
+                secret_obj = ref_controller[token_path]
+                if awskey != secret_obj.key:
                     if args.validate_targets:
                         logger.info("%s key mismatch", token_path)
                         ret_code = 1
                     else:
-                        secret_obj.update_key(key)
+                        secret_obj.update_key(awskey)
                         ref_controller[token_path] = secret_obj
 
             else:

kapitan/refs/base.py

@@ -365,6 +365,9 @@ def _get_backend(self, type_name):
             elif type_name == 'gkms':
                 from kapitan.refs.secrets.gkms import GoogleKMSBackend
                 self.register_backend(GoogleKMSBackend(self.path))
+            elif type_name == 'awskms':
+                from kapitan.refs.secrets.awskms import AWSKMSBackend
+                self.register_backend(AWSKMSBackend(self.path))
             else:
                 raise RefBackendError('no backend for ref type: {}'.format(type_name))
         return self.backends[type_name]
@@ -516,7 +519,7 @@ def __init__(self, data):
 
 def search_target_token_paths(target_secrets_path, targets):
     """
-    returns dict of target and their secret token paths (e.g ?{[gpg/gkms]:mysql/root/password}) in target_secrets_path
+    returns dict of target and their secret token paths (e.g ?{[gpg/gkms/awskms]:mysql/root/password}) in target_secrets_path
     targets is a set of target names used to lookup targets in target_secrets_path
     """
     target_files = defaultdict(list)