End to end encryption of traffic with customer provided certs

[jamsajones]

No description provided.

[mhausenblas]

What is the relation of this issue with #34?

[bcelenza]

@mhausenblas #34 is an extension of this. This issue is for enabling TLS between services with customer provided server certificates, #34 is for adding authorization (and client certificates) for mTLS.

As an aside, this issue will be partially resolved with #39, but will be kept open to pursue other means of accepting customer certificates directly in App Mesh.

[bcelenza]

A proposal has been recently added to #39 which will allow you to import a certificate to AWS Certificate Manager (see Importing Certificates into AWS Certificate Manager) and use it with App Mesh.

If that particular approach does not fit your use cases, I'd love to hear more about where you store your certificates and how you would want App Mesh to use them.

[bcelenza]

You can now try out TLS in App Mesh by importing your certificates into ACM. Check out the walkthrough to get started, and the docs for more info.

We're cooking up a few other options for providing certificates to App Mesh, but in the meantime, we'd love to hear your feedback on importing certificates in ACM as one way to provide your own certs for App Mesh.

[shubharao]

We have identified some potential issues with this feature and therefore disabling it until we investigate further. We will keep you updated about the progress. Apologies!

[bcelenza]

We're working on researching options for providing certificates to Envoy with App Mesh. Here's what we're considering:

1. Retrieve a certificate and private key from the local file system.
2. Retrieve a certificate validation chain from the local file system.
3. Retrieve certificates/private keys and validation chains from an external Secret Discovery Service endpoint (local unix domain socket, or remote address).

For 1 and 2, this will provide part of the solution for enabling SPIFFE/Spire support (#68, #34). The remaining work tackled in that issue will be for providing client certificates to configured backends, and providing extra validation options for verifying certificate information (i.e. SANs for allowed SVIDs).

For customer-provided certificates, what else should we be considering? Would love to hear feedback if you're interested in this particular approach to configuring TLS. Let us know!

[bcelenza]

Hey all, here is our proposal for enabling customer provided certificates. I'll be separating this proposal into three parts:

1. New certificate sources
2. Client policies for TLS (validation context)
3. Client policy defaults

New Certificate Sources

In #39 we're adding the ability to source a Virtual Node's certificate from AWS Certificate Manager (ACM). In this proposal, we'll be adding two new possible sources for a certificate. These are:

1. The local file system
2. A local Secret Discovery Service endpoint hosted via a unix domain socket

When configuring a certificate for a Virtual Node, you can only choose one of the available sources. Each new option is discussed in detail below.

Local File System

A common pattern for binding a TLS certificate in a service mesh is to derive the certificate from the file system the service is running on. In this pattern, external processes (sidecars, injectors, etc.) place and rotate the certificate on the local disk, and the proxy is given access to that certificate. When the certificates content change, the proxy re-binds the certificate from the source.

In App Mesh, you'll be able to create a Virtual Node that references a path on the local file system for the certificate materials. Here's an example call:

$ aws appmesh create-virtual-node --mesh-name my-mesh \
    --virtual-node-name my-node \
    --spec
{
    "listeners": [{
        "portMapping": {
            "port": 443,
            "protocol": "http"
        },
        "tls": {
            "mode": "STRICT",
            "certificate": {
                "file": {
                    // (REQUIRED) The path to the certificate chain.
                    "certificateChain": "/path/to/cert-chain.pem",
                    // (REQUIRED) The path to the private key.
                    "privateKey": "/path/to/private-key.pem"
                }
            } 
        }
    }],
    "serviceDiscovery": {
        "dns": {
            "hostname": "my-node.my-mesh.local"
        }
    }
}

Secret Discovery Service via Unix Domain Socket

An emerging pattern for TLS certificate binding in service mesh is through the use of the Universal Data Plane API's Secret Discovery Service. For certificates derived from AWS Certificate Manager, App Mesh hosts a secure SDS endpoint which delivers the certificates and validation context.

This option adds the ability for the proxy to connect to a local process (i.e. sidecar) which is hosting an SDS endpoint via a Unix Domain Socket (UDS). Here's an example reference to a certificate stored as a secret in a local SDS endpoint:

$ aws appmesh create-virtual-node --mesh-name my-mesh \
    --virtual-node-name my-node \
    --spec
{
    "listeners": [{
        "portMapping": {
            "port": 443,
            "protocol": "http"
        },
        "tls": {
            "mode": "STRICT",
            "certificate": {
                "sds": {
                    // (REQUIRED) The name of the secret to fetch
                    "secretName": "my-node-certificate",
                    // (REQUIRED) Where to fetch it from
                    "source": {
                        "unixDomainSocket": {
                            // (REQUIRED) The path to the unix domain socket.
                            "path": "/path/to/sds-server.sock"
                        }
                    }
                }
            } 
        }
    }],
    "serviceDiscovery": {
        "dns": {
            "hostname": "my-node.my-mesh.local"
        }
    }
}

Client Policies

Enforcing the use of TLS, and validating the certificate from the upstream service, is an important aspect of a zero-trust network. Using these patterns allows you to ensure that connectivity is secured and the service you're talking to is authentic (trusted).

In this proposal we're adding the ability to specify a policy on a Virtual Node backend to set various settings from the perspective of the client. We're calling this new policy simply a ClientPolicy. The first phase will add settings specific to TLS origination and validation, but the policy will be expanded over time to include other settings pertinent to the client's perspective.

For TLS-specific settings, you will be able to specify whether or not you want TLS enforced (i.e. a negotiation is attempted, and connection fails if it cannot), optionally what ports you want to enforce it on, and the certificate authorities to use when validating the certificate. For the certificate authorities to use, you'll have three options to choose from:

1. A set of ACM Private Certificate Authorities
2. A reference to the local file system
3. A reference to a local Secret Discovery Service over a unix domain socket

We'll use ACM as a straight forward example of how to configure the TLS client policy.

Enforcing TLS with an ACM Certificate Authority ARN

The follow example shows a client policy set on a Virtual Node's backend to a Virtual Service, which enforces the use of TLS on port 443 only.

$ aws appmesh create-virtual-node --mesh-name my-mesh \
    --virtual-node-name my-node \
    --spec
{
    "backends": [
        {
            "virtualService": {
                "virtualServiceName": "foo.mesh.local",
                // (OPTIONAL) Client policy for this backend.
                "clientPolicy": {
                    // (OPTIONAL) New struct for specifying backend TLS settings
                    "tls": {
                        // (OPTIONAL) When true, enforces the use of TLS for this backend.
                        // Default: true
                        "enforce": true,
                        // (OPTIONAL) Scope down which upstream ports to enforce TLS on.
                        // Default: all ports
                        "ports": [443],
                        // (REQUIRED) Validation context for TLS connections to this backend.
                        "validation": {
                            // (REQUIRED) Settings for determining where to retrieve the chain of trust.
                            "trust": {
                                "acm": {
                                    // (REQUIRED) The ARNs of trusted CAs.
                                    "certificateAuthorityArns": [
                                        "arn:aws:acm-pca:us-west-2:123456789012:certificate-authority/17d7df97-2564-4d44-93a8-3aed69f5b294"
                                    ]
                                }
                            }
                        }
                    }
                }
            }
        }
    ]
}

Note that in the above example, both enforce and ports in the client policy's TLS settings are optional. If you don't specify them, TLS will be enforced on all ports by default. You can specify up to 10 ACM certificate authority ARNs to use when validating the upstream certificate. If one of the certificate authorities has signed the upstream certificate, then it is considered valid.

Enforcing TLS with a certificate chain on the local file system

A common pattern for enforcing TLS is to use the default chain of trust on the local file system (or some other locally stored chain) to validate the upstream service's certificate. Most operating system ship with a bundle of root certificate authorities that can be used to validate public services.

Here's an example of a policy which enforces TLS on port 443 using the local file system to derive the certificate authorities that will be used to validate the upstream service's certificate:

$ aws appmesh create-virtual-node --mesh-name my-mesh \
    --virtual-node-name my-node \
    --spec
{
    "backends": [
        {
            "virtualService": {
                "virtualServiceName": "foo.mesh.local",
                "clientPolicy": {
                     // (OPTIONAL) New struct for specifying backend TLS settings
                    "tls": {
                        // (OPTIONAL) When true, enforces the use of TLS for this backend.
                        // Default: true
                        "enforce": true,
                        // (OPTIONAL) Scope down which upstream ports to enforce TLS on.
                        // Default: all ports
                        "ports": [443],
                        // (REQUIRED) Validation context for TLS connections to this backend.
                        // If enforce is set to true, either this or validation in backend defaults must be set.
                        "validation": {
                            // (REQUIRED) Settings for determining where to fetch the chain of trust from.
                            "trust": {
                                "file": {
                                    // (REQUIRED) The path to the certificate chain.
                                    "certificateChain": "/path/to/cert-chain.pem"
                                }
                            }
                        }
                    }
                }
            }
        }
    ]
}

Enforcing TLS with a certificate chain from a local Secret Discovery Service

Much like sourcing certificates from a Secret Discovery Service hosted over a local UDS endpoint, you'll be able to do the same for the trusted certificate authorities on a client policy:

$ aws appmesh create-virtual-node --mesh-name my-mesh \
    --virtual-node-name my-node \
    --spec
{
    "backends": [
        {
            "virtualService": {
                "virtualServiceName": "foo.mesh.local",
                "clientPolicy": {
                    // (OPTIONAL) New struct for specifying backend TLS settings
                    "tls": {
                        // (OPTIONAL) When true, enforces the use of TLS for this backend.
                        // Default: true
                        "enforce": true,
                        // (OPTIONAL) Scope down which upstream ports to enforce TLS on.
                        // Default: all ports
                        "ports": [443],
                        // (REQUIRED) Validation context for TLS connections to this backend.
                        // If enforce is set to true, either this or validation in backend defaults must be set.
                        "validation": {
                            // (REQUIRED) Settings for determining where to fetch the chain of trust from.
                            "trust": {
                                "sds": {
                                    // (REQUIRED) The name of the secret to fetch
                                    "secretName": "foo-cert-chain",
                                    // (REQUIRED) Where to fetch it from
                                    "source": {
                                        "unixDomainSocket": {
                                            // (REQUIRED) The path to the unix domain socket.
                                            "path": "/path/to/sds-server.sock"
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    ]
}

Client Policy Defaults

Finally, we recognize that while in-line policies provide a lot of flexibility in how you can enforce TLS on a backend, many policies will be redundant, with occasions where a single policy could be specified for everything.

With that in mind, we're adding the ability to specify default settings for backends on a Virtual Node. With this new field, you'll be able to setup a broad policy covering most (if not all) of your backends with the option to override settings on a per-backend basis.

Let's start with a basic introduction to the new field. This new field will be at the top-level of a Virtual Node's specification, and contain the exact same settings discussed in Client Policies above:

$ aws appmesh create-virtual-node --mesh-name foo \
    --virtual-node-name my-node \
    --spec
{
    "backends": [
        {
            "virtualService": {
                "virtualServiceName": "foo.mesh.local"
            }
        },
        {
            "virtualService": {
                "virtualServiceName": "bar.mesh.local"
            }
        },
        {
            "virtualService": {
                "virtualServiceName": "baz.mesh.local"
            }
        }
    ],
    // (OPTIONAL) Specify options to apply to all backends.
    // The following defaults are applied, but can be overriden
    // on an individual backend basis when required.
    "backendDefaults": {
        "clientPolicy": {
            "tls": {
                // (OPTIONAL) When true, enforces the use of TLS for this backend.
                // Default: true
                "enforce": true,
                // (OPTIONAL) Scope down which upstream ports to enforce TLS on.
                // Default: all ports
                "ports": [443],
                // (REQUIRED) Validation context for TLS connections to this backend.
                // If enforce is set to true, either this or validation in backend defaults must be set.
                "validation": {
                    // (REQUIRED) Settings for determining where to fetch the chain of trust from.
                    "trust": {
                        "acm": {
                            // (REQUIRED) The ARNs of trusted CAs.
                            "certificateAuthorityArns": [
                                "arn:aws:acm-pca:us-west-2:123456789012:certificate-authority/17d7df97-2564-4d44-93a8-3aed69f5b294"
                            ]
                        }
                    }
                }
            }
        }
    }
}

In the above example, TLS is enforced on port 443 for all backends (foo, bar, and baz) using a certificate authority from ACM to validate the upstream services certificates.

However, what if we didn't want to enforce TLS for bar.mesh.local? We can simply override that setting on the backend directly:

$ aws appmesh create-virtual-node --mesh-name foo \
    --virtual-node-name my-node \
    --spec
{
    "backends": [
        {
            "virtualService": {
                "virtualServiceName": "foo.mesh.local"
            }
        },
        {
            "virtualService": {
                "virtualServiceName": "bar.mesh.local",
                "clientPolicy": {
                    "tls": {
                        // Overridden setting to disable TLS enforcement for this backend
                        "enforce": false
                    }
                }
            }
        },
        {
            "virtualService": {
                "virtualServiceName": "baz.mesh.local"
            }
        }
    ],
    // The following defaults are applied, but can be overriden
    // on an individual backend basis when required.
    "backendDefaults": {
        "clientPolicy": {
            "tls": {
                "enforce": true,
                // Default: all ports
                "ports": [443],
                "validation": {
                    "trust": {
                        "acm": {
                            "certificateAuthorityArns": [
                                "arn:aws:acm-pca:us-west-2:123456789012:certificate-authority/17d7df97-2564-4d44-93a8-3aed69f5b294"
                            ]
                        }
                    }
                }
            }
        }
    }
}

Similarly, what if we needed to specify a separate bundle of certificate authorities to validate the certificate for the backend baz.mesh.local? We could do that as follows:

$ aws appmesh create-virtual-node --mesh-name foo \
    --virtual-node-name my-node \
    --spec
{
    "backends": [
        {
            "virtualService": {
                "virtualServiceName": "foo.mesh.local"
            }
        },
        {
            "virtualService": {
                "virtualServiceName": "bar.mesh.local"
            }
        },
        {
            "virtualService": {
                "virtualServiceName": "baz.mesh.local",
                // Overridden TLS settings to change CAs
                "clientPolicy": {
                    "tls": {
                        "validation": {
                            "trust": {
                               "file": {
                                    "certificateChain": "/path/to/cert-chain.pem"
                                }
                            }
                        }
                    }
                }
            }
        }
    ],
    // The following defaults are applied, but can be overriden
    // on an individual backend basis when required.
    "backendDefaults": {
        "clientPolicy": {
            "tls": {
                "enforce": true,
                // Default: all ports
                "ports": [443],
                "validation": {
                    "trust": {
                        "acm": {
                            "certificateAuthorityArns": [
                                "arn:aws:acm-pca:us-west-2:123456789012:certificate-authority/17d7df97-2564-4d44-93a8-3aed69f5b294"
                            ]
                        }
                    }
                }
            }
        }
    }
}

In the above example, ACM PCA will be used to validate certificates for all backends with the exception of baz.mesh.local, which will use the local file system.

Summary

We hope these changes fit what you're looking for and provide the basis for many other types of TLS certificate providers. We'll be updating this issue once we have the feature enabled in our Preview Channel. Until then, we'd love to hear your feedback on this proposal in the comments.

Thanks!

[joshuabaird]

@bcelenza As always, the detailed description of these new features is very much appreciated. It's very helpful to my team as we continue to plan and design our App Mesh deployment. We're looking forward to the "client policy" features described here.

[dfezzie]

Hey everyone! We just enabled client policy support and TLS termination using file based certificates in preview. There is a walk through of this new feature here: https://github.com/aws/aws-app-mesh-examples/tree/master/walkthroughs/howto-tls-file-provided

As always, feel free to leave any feedback you might have. We are still working towards SDS provided end to end encryption. Stay tuned for more updates on that at a later date.

[bcelenza]

TLS with customer-provided certificates is now generally available in the App Mesh APIs, SDKs, and CloudFormation for all regions that App Mesh operates in. Check out the latest user guide for more information.

Please note that at this time the App Mesh console experience has not been updated. Additionally, support in the Kubernetes controller for App Mesh is pending merge and release (see this PR for the latest).

We'll be holding this issue open until everything is closed out, after which a more formal announcement will be made.

A huge thanks to all who have provided feedback to us through the design and preview period for the feature.

[bcelenza]

Kubernetes controller changes for TLS are now available in release v0.4.0. See the release notes for the full list of changes and improvements we've made in this release.

[bcelenza]

The AWS console now also supports TLS in App Mesh.

You can read more about the release in this blog post.

We're closing this issue, but please check out these additional roadmap items for TLS support:

1. Allowing the application to optionally originate TLS instead of the proxy: Feature Request: TLS negotiation between the downstream application and upstream service #162
2. Support for revocation lists from ACM PCA: Feature Request: Allow the use of ACM PCA CRLs when using TLS #172