Skip to content

Commit

Permalink
Automatically Generate and Use Gossip Encryption Key (#738)
Browse files Browse the repository at this point in the history
* Add global.gossipEncryption.autogenerate to values.yaml

* Add description of key autogen

* Add initial autogen-encryption-job yaml

* Fix run condition on autogen

* autogen is false by default

* Set secretName/Key correctly, flesh out curling k8s a bit

* Add some basic bats tests

* Add tests for user set values for secretName and secretKey

* Beef up comment for gossip encryption

* Add notes

* Update values.yaml to include autoGenerate

* Add gossip-encryption-autogen-job.yaml

* Port over autogen-encryption-job to gossip-encryption-autogen-job

* Rename autogen-encryption-job.bats to gossip-encryption-autogen-job.bats

* Remove change made to statefulset

* Add check that secretName and secretKey are not set

* Fix test failures

* Set GOSSIP_KEY properly in server statefulset

* Remove setting secretName and secretKey for autogen

* Check if secret exists via 200 resp

* Add gossip autogen to client daemonset

* Send the key to secrets

* Base64 encrypt consul key

* Add podsecuritypolicy, role, rolebinding, and SA

* Rename -gossip-encryption-autogen to -gossip-encryption-autogenerate

* Rename *-autogen-* to *-autogeneration-*

* Remove text about respecting user set secretName and secretKey for autogen

* Rename gossip-encryption-autogen to gossip-encryption-autogeneration

* Add some great tests!

* Update changelog

* Fix filename reference in job bats file

* Update charts/consul/templates/gossip-encryption-autogeneration-job.yaml

Co-authored-by: Iryna Shustava <ishustava@users.noreply.github.com>

* Update charts/consul/test/unit/gossip-encryption-autogeneration-job.bats

Co-authored-by: Iryna Shustava <ishustava@users.noreply.github.com>

* Don't do the pre-check, but don't replace the current secret

* Update charts/consul/test/unit/gossip-encryption-autogeneration-job.bats

Co-authored-by: Iryna Shustava <ishustava@users.noreply.github.com>

* Only give role create and get perms

* Update charts/consul/test/unit/gossip-encryption-autogeneration-podsecurity.bats

Co-authored-by: Iryna Shustava <ishustava@users.noreply.github.com>

* Update charts/consul/values.yaml

Co-authored-by: Iryna Shustava <ishustava@users.noreply.github.com>

* Return kubectl command to what it was

* Test that GOSSIP_KEY gets passed in on the encrypt flag

* Autogen job does not run as root

* Add check that gossip encryption is getting set

* Fix test for gossip encryption in acceptance tests

* Update charts/consul/test/acceptance/tests/basic/basic_test.go

Co-authored-by: Ashwin Venkatesh <ashwin@hashicorp.com>

* Rename s/autogeneration/autogenerate/ for gossip-encryption-autogeneration-*

* Use correct filename in bats

* Rename podsecuritypolicy test to be correct

* Change v1 to metaV1

* Update charts/consul/test/acceptance/tests/basic/basic_test.go

Co-authored-by: Luke Kysow <1034429+lkysow@users.noreply.github.com>

* Test the other keyring value

* Remove arbitrary test

* Remove document separator

* Remove temp dir from job

* Remove get perm from autogenerate role

* Add -encrypt flag test for client-daemonset

* Change autogen to a feature in the CHANGELOG

* Update comment on values.yaml

* Update charts/consul/test/acceptance/tests/basic/basic_test.go

Co-authored-by: Ashwin Venkatesh <ashwin@hashicorp.com>

Co-authored-by: Iryna Shustava <ishustava@users.noreply.github.com>
Co-authored-by: Ashwin Venkatesh <ashwin@hashicorp.com>
Co-authored-by: Luke Kysow <1034429+lkysow@users.noreply.github.com>
  • Loading branch information
4 people authored Oct 1, 2021
1 parent a376f18 commit 237e02e
Show file tree
Hide file tree
Showing 17 changed files with 478 additions and 23 deletions.
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
## UNRELEASED

FEATURES:
* Helm Chart
* Add automatic generation of gossip encryption with `global.gossipEncryption.autoGenerate=true`. [[GH-738](https://github.com/hashicorp/consul-k8s/pull/738)]

IMPROVEMENTS:
* Control Plane
* Upgrade Docker image Alpine version from 3.13 to 3.14. [[GH-737](https://github.com/hashicorp/consul-k8s/pull/737)]
Expand Down
9 changes: 7 additions & 2 deletions charts/consul/templates/client-daemonset.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -168,12 +168,17 @@ spec:
fieldPath: status.podIP
- name: CONSUL_DISABLE_PERM_MGMT
value: "true"
{{- if (and .Values.global.gossipEncryption.secretName .Values.global.gossipEncryption.secretKey) }}
{{- if (or .Values.global.gossipEncryption.autoGenerate (and .Values.global.gossipEncryption.secretName .Values.global.gossipEncryption.secretKey)) }}
- name: GOSSIP_KEY
valueFrom:
secretKeyRef:
{{- if .Values.global.gossipEncryption.autoGenerate }}
name: {{ template "consul.fullname" . }}-gossip-encryption-key
key: key
{{- else if (and .Values.global.gossipEncryption.secretName .Values.global.gossipEncryption.secretKey) }}
name: {{ .Values.global.gossipEncryption.secretName }}
key: {{ .Values.global.gossipEncryption.secretKey }}
{{- end }}
{{- end }}
{{- if (and .Values.server.enterpriseLicense.secretName .Values.server.enterpriseLicense.secretKey .Values.server.enterpriseLicense.enableLicenseAutoload (not .Values.global.acls.manageSystemACLs)) }}
- name: CONSUL_LICENSE_PATH
Expand Down Expand Up @@ -252,7 +257,7 @@ spec:
{{- end }}
-datacenter={{ .Values.global.datacenter }} \
-data-dir=/consul/data \
{{- if (and .Values.global.gossipEncryption.secretName .Values.global.gossipEncryption.secretKey) }}
{{- if (or .Values.global.gossipEncryption.autoGenerate (and .Values.global.gossipEncryption.secretName .Values.global.gossipEncryption.secretKey)) }}
-encrypt="${GOSSIP_KEY}" \
{{- end }}
{{- if .Values.client.join }}
Expand Down
71 changes: 71 additions & 0 deletions charts/consul/templates/gossip-encryption-autogenerate-job.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
{{- if .Values.global.gossipEncryption.autoGenerate }}
{{- if (or .Values.global.gossipEncryption.secretName .Values.global.gossipEncryption.secretKey) }}
{{ fail "If global.gossipEncryption.autoGenerate is true, global.gossipEncryption.secretName and global.gossipEncryption.secretKey must not be set." }}
{{ end }}
# automatically generate encryption key for gossip protocol and save it in Kubernetes secret
apiVersion: batch/v1
kind: Job
metadata:
name: {{ template "consul.fullname" . }}-gossip-encryption-autogenerate
namespace: {{ .Release.Namespace }}
labels:
app: {{ template "consul.name" . }}
chart: {{ template "consul.chart" . }}
heritage: {{ .Release.Service }}
release: {{ .Release.Name }}
annotations:
"helm.sh/hook": pre-install,pre-upgrade
"helm.sh/hook-weight": "1"
"helm.sh/hook-delete-policy": hook-succeeded,before-hook-creation
spec:
template:
metadata:
name: {{ template "consul.fullname" . }}-gossip-encryption-autogenerate
labels:
app: {{ template "consul.name" . }}
chart: {{ template "consul.chart" . }}
release: {{ .Release.Name }}
component: gossip-encryption-autogeneneration
annotations:
"consul.hashicorp.com/connect-inject": "false"
spec:
restartPolicy: Never
serviceAccountName: {{ template "consul.fullname" . }}-gossip-encryption-autogenerate
securityContext:
runAsNonRoot: true
runAsGroup: 1000
runAsUser: 100
fsGroup: 1000
containers:
- name: gossip-encryption-autogen
image: "{{ .Values.global.image }}"
env:
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
# We're using POST requests below to create secrets via Kubernetes API.
# Note that in the subsequent runs of the job, POST requests will
# return a 409 because these secrets would already exist;
# we are ignoring these response codes.
command:
- "/bin/sh"
- "-ec"
- |
secretName={{ template "consul.fullname" . }}-gossip-encryption-key
secretKey=key
keyValue=$(consul keygen | base64)
curl -s -X POST --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}/api/v1/namespaces/${NAMESPACE}/secrets \
-H "Authorization: Bearer $( cat /var/run/secrets/kubernetes.io/serviceaccount/token )" \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-d "{ \"kind\": \"Secret\", \"apiVersion\": \"v1\", \"metadata\": { \"name\": \"${secretName}\", \"namespace\": \"${NAMESPACE}\" }, \"type\": \"Opaque\", \"data\": { \"${secretKey}\": \"${keyValue}\" }}" > /dev/null
resources:
requests:
memory: "50Mi"
cpu: "50m"
limits:
memory: "50Mi"
cpu: "50m"
{{- end }}
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
{{- if .Values.global.gossipEncryption.autoGenerate }}
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: {{ template "consul.fullname" . }}-gossip-encryption-autogenerate
namespace: {{ .Release.Namespace }}
labels:
app: {{ template "consul.name" . }}
chart: {{ template "consul.chart" . }}
heritage: {{ .Release.Service }}
release: {{ .Release.Name }}
annotations:
"helm.sh/hook": pre-install,pre-upgrade
"helm.sh/hook-delete-policy": before-hook-creation
spec:
privileged: false
# Required to prevent escalations to root.
allowPrivilegeEscalation: false
# This is redundant with non-root + disallow privilege escalation,
# but we can provide it for defense in depth.
requiredDropCapabilities:
- ALL
# Allow core volume types.
volumes:
- 'secret'
hostNetwork: false
hostIPC: false
hostPID: false
runAsUser:
rule: 'RunAsAny'
seLinux:
rule: 'RunAsAny'
supplementalGroups:
rule: 'RunAsAny'
fsGroup:
rule: 'RunAsAny'
readOnlyRootFilesystem: false
{{- end }}
30 changes: 30 additions & 0 deletions charts/consul/templates/gossip-encryption-autogenerate-role.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
{{- if .Values.global.gossipEncryption.autoGenerate }}
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: {{ template "consul.fullname" . }}-gossip-encryption-autogenerate
namespace: {{ .Release.Namespace }}
labels:
app: {{ template "consul.name" . }}
chart: {{ template "consul.chart" . }}
heritage: {{ .Release.Service }}
release: {{ .Release.Name }}
annotations:
"helm.sh/hook": pre-install,pre-upgrade
"helm.sh/hook-delete-policy": before-hook-creation
rules:
- apiGroups: [""]
resources:
- secrets
verbs:
- create
{{- if .Values.global.enablePodSecurityPolicies }}
- apiGroups: ["policy"]
resources:
- podsecuritypolicies
verbs:
- use
resourceNames:
- {{ template "consul.fullname" . }}-gossip-encryption-autogenerate
{{- end }}
{{- end }}
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{{- if .Values.global.gossipEncryption.autoGenerate }}
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: {{ template "consul.fullname" . }}-gossip-encryption-autogenerate
namespace: {{ .Release.Namespace }}
labels:
app: {{ template "consul.name" . }}
chart: {{ template "consul.chart" . }}
heritage: {{ .Release.Service }}
release: {{ .Release.Name }}
annotations:
"helm.sh/hook": pre-install,pre-upgrade
"helm.sh/hook-delete-policy": before-hook-creation
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: {{ template "consul.fullname" . }}-gossip-encryption-autogenerate
subjects:
- kind: ServiceAccount
name: {{ template "consul.fullname" . }}-gossip-encryption-autogenerate
{{- end }}
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{{- if .Values.global.gossipEncryption.autoGenerate }}
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ template "consul.fullname" . }}-gossip-encryption-autogenerate
namespace: {{ .Release.Namespace }}
labels:
app: {{ template "consul.name" . }}
chart: {{ template "consul.chart" . }}
heritage: {{ .Release.Service }}
release: {{ .Release.Name }}
annotations:
"helm.sh/hook": pre-install,pre-upgrade
"helm.sh/hook-delete-policy": before-hook-creation
{{- with .Values.global.imagePullSecrets }}
imagePullSecrets:
{{- range . }}
- name: {{ .name }}
{{- end }}
{{- end }}
{{- end }}
9 changes: 7 additions & 2 deletions charts/consul/templates/server-statefulset.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -156,12 +156,17 @@ spec:
fieldPath: metadata.namespace
- name: CONSUL_DISABLE_PERM_MGMT
value: "true"
{{- if (and .Values.global.gossipEncryption.secretName .Values.global.gossipEncryption.secretKey) }}
{{- if (or .Values.global.gossipEncryption.autoGenerate (and .Values.global.gossipEncryption.secretName .Values.global.gossipEncryption.secretKey)) }}
- name: GOSSIP_KEY
valueFrom:
secretKeyRef:
{{- if .Values.global.gossipEncryption.autoGenerate }}
name: {{ template "consul.fullname" . }}-gossip-encryption-key
key: key
{{- else if (and .Values.global.gossipEncryption.secretName .Values.global.gossipEncryption.secretKey) }}
name: {{ .Values.global.gossipEncryption.secretName }}
key: {{ .Values.global.gossipEncryption.secretKey }}
{{- end }}
{{- end }}
{{- if .Values.global.tls.enabled }}
- name: CONSUL_HTTP_ADDR
Expand Down Expand Up @@ -223,7 +228,7 @@ spec:
-datacenter={{ .Values.global.datacenter }} \
-data-dir=/consul/data \
-domain={{ .Values.global.domain }} \
{{- if (and .Values.global.gossipEncryption.secretName .Values.global.gossipEncryption.secretKey) }}
{{- if (or .Values.global.gossipEncryption.autoGenerate (and .Values.global.gossipEncryption.secretName .Values.global.gossipEncryption.secretKey)) }}
-encrypt="${GOSSIP_KEY}" \
{{- end }}
{{- if .Values.server.connect }}
Expand Down
28 changes: 25 additions & 3 deletions charts/consul/test/acceptance/tests/basic/basic_test.go
Original file line number Diff line number Diff line change
@@ -1,15 +1,18 @@
package basic

import (
"context"
"fmt"
"strconv"
"strings"
"testing"

"github.com/hashicorp/consul-k8s/charts/consul/test/acceptance/framework/consul"
"github.com/hashicorp/consul-k8s/charts/consul/test/acceptance/framework/helpers"
"github.com/hashicorp/consul-k8s/charts/consul/test/acceptance/framework/logger"
"github.com/hashicorp/consul/api"
"github.com/stretchr/testify/require"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// Test that the basic installation, i.e. just
Expand Down Expand Up @@ -39,9 +42,10 @@ func TestBasicInstallation(t *testing.T) {
t.Run(name, func(t *testing.T) {
releaseName := helpers.RandomName()
helmValues := map[string]string{
"global.acls.manageSystemACLs": strconv.FormatBool(c.secure),
"global.tls.enabled": strconv.FormatBool(c.secure),
"global.tls.enableAutoEncrypt": strconv.FormatBool(c.autoEncrypt),
"global.acls.manageSystemACLs": strconv.FormatBool(c.secure),
"global.tls.enabled": strconv.FormatBool(c.secure),
"global.gossipEncryption.autoGenerate": strconv.FormatBool(c.secure),
"global.tls.enableAutoEncrypt": strconv.FormatBool(c.autoEncrypt),
}
consulCluster := consul.NewHelmCluster(t, helmValues, suite.Environment().DefaultContext(t), suite.Config(), releaseName)

Expand All @@ -63,6 +67,24 @@ func TestBasicInstallation(t *testing.T) {
kv, _, err := client.KV().Get(randomKey, nil)
require.NoError(t, err)
require.Equal(t, kv.Value, randomValue)

// Check that autogenerated gossip encryption key is being used
if c.secure {
secretName := fmt.Sprintf("%s-consul-gossip-encryption-key", releaseName)
secretKey := "key"

keyring, err := client.Operator().KeyringList(nil)
require.NoError(t, err)

testContext := suite.Environment().DefaultContext(t)
secret, err := testContext.KubernetesClient(t).CoreV1().Secrets(testContext.KubectlOptions(t).Namespace).Get(context.Background(), secretName, metav1.GetOptions{})
require.NoError(t, err)
gossipEncryptionKey := strings.TrimSpace(string(secret.Data[secretKey]))

require.Len(t, keyring, 2)
require.Contains(t, keyring[0].Keys, gossipEncryptionKey)
require.Contains(t, keyring[1].Keys, gossipEncryptionKey)
}
})
}
}
21 changes: 21 additions & 0 deletions charts/consul/test/unit/client-daemonset.bats
Original file line number Diff line number Diff line change
Expand Up @@ -606,6 +606,27 @@ load _helpers
[ "${actual}" = "" ]
}

@test "client/DaemonSet: gossip encryption autogeneration properly sets secretName and secretKey" {
cd `chart_dir`
local actual=$(helm template \
-s templates/client-daemonset.yaml \
--set 'global.gossipEncryption.autoGenerate=true' \
. | tee /dev/stderr |
yq '.spec.template.spec.containers[] | select(.name=="consul") | .env[] | select(.name == "GOSSIP_KEY") | .valueFrom.secretKeyRef | [.name=="RELEASE-NAME-consul-gossip-encryption-key", .key="key"] | all' | tee /dev/stderr)
[ "${actual}" = "true" ]
}

@test "client/DaemonSet: gossip encryption key is passed in via the -encrypt flag" {
cd `chart_dir`
local actual=$(helm template \
-s templates/client-daemonset.yaml \
--set 'global.gossipEncryption.autoGenerate=true' \
. | tee /dev/stderr |
yq '.spec.template.spec.containers[] | select(.name=="consul") | .command | any(contains("-encrypt=\"${GOSSIP_KEY}\""))'
| tee /dev/stderr)
[ "${actual}" = "true" ]
}

@test "client/DaemonSet: gossip encryption disabled in client DaemonSet when secretName is missing" {
cd `chart_dir`
local actual=$(helm template \
Expand Down
Loading

0 comments on commit 237e02e

Please sign in to comment.