secret-agent - Secret generator and manager for k8s

The secret-agent is a Kubernetes operator that generates the secrets required by the ForgeRock® Identity Platform. The secrets are stored in-cluster as Kubernetes secrets and can also be stored in a cloud secret manager.

Roadmap

Secret agent is now feature complete. Future updates will be limited to bug fixes.

Secret agent was originally designed to fulfill a short term obective to create and manage secrets for the ForgeRock platform running on Kubernetes. The longer term roadmap for platform secrets management is centered around:

• Direct integration with cloud secret management (GCP Secret Manager, Vault, etc.) using ForgeRock's commons secrets API.
• Management of certificates using https://certmanager.io. Cert-manager is the defacto Kubernetes standard for certificate management.
• Backup and restore of Kubernetes secrets using cloud native technologies such as https://velero.io, or vendor provided backup services (Backup for GKE).

Installation

To install the latest secret-agent release in a Kubernetes environment, run:

kubectl apply -f https://github.com/ForgeRock/secret-agent/releases/latest/download/secret-agent.yaml

Specific versions of the operator can be installed by running:

SA_VERSION=v0.1.0 kubectl apply -f https://github.com/ForgeRock/secret-agent/releases/download/${SA_VERSION}/secret-agent.yaml

Configuration

Once the operator is installed, new secrets can be generated by providing a secret agent configuration (SAC) object. The SAC is a custom kubernetes object monitored by the secret-agent operator. All the secrets’ specifications are defined through the SAC.

For more information on how to create a SAC, see the Secret Agent Configuration Schema and/or the Examples sections.

Once the SAC file has been created, it can be pushed to the cluster as with any other resource.

For example:

kubectl create -f config/samples/secret-agent_v1alpha1_secretagentconfiguration.yaml

It is important to note that the Kubernetes secrets produced by the secret-agent will be placed in the same namespace as the SAC. If similar secrets are desired in multiple namespaces, one SAC would be required per namespace.

Usage

Enabling Cloud Backup

The secret-agent can be configured to back up all the generated secrets in a cloud provider's secret manager solution. When this feature is enabled, secrets stored in the secret managers are considered the source of truth.

If a cloud provider has been configured, the operator will attempt to load the secret data from that cloud provider. If the secret is found in the cloud provider's secret manager, the operator will use the found data as the Kubernetes secret data. The operator will only generate new secrets if no secret data is found in the cloud provider.

The secret-agent supports the following cloud providers:

• Google Secret Manager
• AWS Secrets Manager
• Azure Key Vault

It is possible to run the secret-agent without setting up a cloud provider. This is useful when debugging or testing applications. To disable cloud provider support, set spec.appConfig.secretsManager to “none”. This is only possible if spec.appConfig.createKubernetesObjects is set to true.

In addition, it is possible to configure the secret-agent to store secrets in the secret manager without creating local Kubernetes secrets. This is useful if your applications can access the cloud secret manager directly and the secret-agent is only used to generate such secrets. To achieve this, set spec.appConfig.createKubernetesObjects to false. Do note that spec.appConfig.secretsManager cannot be set to "none".

In order to fetch and store secrets in the AWS Secrets Manager, the user must provide credentials with the necessary permissions.

Set up Cloud Backup With AWS Secret Manager

The secret-agent expects credentials to be discoverable via standard AWS mechanisms. These credentials can be provided in a number of ways as per the link.

However, when running inside AWS the preferred method should be to attach a serviceAccount to the deployment with properly scope policies on the role.

• Environment Variables:
  • AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN
  • AWS_WEB_IDENTITY_TOKEN_FILE -> this will be handled by the IAM controller for each deployment correctly set up annotations on the serviceAccount.
• Shared Credentials file: ~/.aws/credentials
• Shared Configuration file: ~/.aws/config
• EC2 Instance Metadata_v2: Obtains credentials from 169.254.169.254
  • This is not ideal as all the pods on the node will have access to the same policy, this may not be a desired outcome if topologySkew are not properly segragating workloads across different nodes/node pools.

Refer to AWS documentation for instructions on how to obtain credentials and grant necessary permissions to access the AWS Secrets Manager. The secret-agent needs to access read/write secrets. This can be achieved by allowing access to the arn:aws:iam::aws:policy/SecretsManagerReadWrite AWS managed policy.

When running outside of AWS environments it is possible to provide custom credentials via a Kubernetes secret. The secret reference is provided in the SAC in spec.appConfig.credentialsSecretName. In the default secret-agent deployment, the user is expected to publish the cloud credentials' secret in the same namespace as the operator. This target namespace can be changed by changing the runtime argument --cloud-secrets-namespace=[NS_NAME] located in the operator's manifest. If this argument is omitted completely, the namespace will default to the namespace of each SAC.

Once these credentials are posted to a Kubernetes secret, the next step is to configure the AWS Secret Manager using the SecretAgentConfiguration.

For example, the following configuration targets AWS Secret Manager in us-east-1:

apiVersion: secret-agent.secrets.forgerock.io/v1alpha1
kind: SecretAgentConfiguration
metadata:
  name: standard-forgerock-example
  namespace: test-sa
spec:
  appConfig:
    createKubernetesObjects: true
    credentialsSecretName: cloud-credentials [** optional**]
    secretsManager: AWS
    awsRegion: us-east-1

optional for AWS: The cloud-credentials secret referenced in spec.appConfig.credentialsSecretName would look like this:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: cloud-credentials
  namespace: test-sa
data:
  AWS_ACCESS_KEY_ID: QU....[base64 encoded key].....GY=
  AWS_SECRET_ACCESS_KEY: cRB.....[base64 encoded access key].......BB==

Note: The maximum secret size supported by AWS is 65Kb For more information, see AWS documentation.

Set up Cloud Backup With GCP Secret Manager

The secret-agent expects credentials to be discoverable via standard GCP mechanisms. These credentials can be provided in a number of ways, including:

• Workload identity (recommended for GKE deployments)
• Credentials file with user accounts or service accounts.

Please refer to the GCP Documentation for instructions on how to create a service account with the necessary permissions to access the GCP Secrets Manager. The secret-agent needs access to read/write secrets. This can be achieved by assigning the Secret Manager Admin role to the service account provided.

Workload identity

Workload Identity is the recommended way to access Google Cloud services from applications running within GKE. For more information on how to enable workload identity see GCP Documentation.

In general, the user creates a Google Cloud service account with the proper role attached and enables workload identity in their GKE cluster. The Kubernetes services account is already created for you during the secret-agent deployment.

Run the following commands to enable workload identity for the secret-agent deployment:

PROJECTID=myproject #GCP project ID
GSA_NAME=mygcpserviceaccount #GCP service account name
# Create the GCP IAM policy binding
gcloud iam service-accounts add-iam-policy-binding --role roles/iam.workloadIdentityUser --member "serviceAccount:${PROJECTID}.svc.id.goog[secret-agent-system/secret-agent-controller-manager]" ${GSA_NAME}@${PROJECTID}.iam.gserviceaccount.com
# Annotate the Kubernetes service account
kubectl -n secret-agent-system annotate serviceaccounts secret-agent-controller-manager iam.gke.io/gcp-service-account=${GSA_NAME}@${PROJECTID}.iam.gserviceaccount.com

Note: in order to use workload identity, no spec.appConfig.credentialsSecretName should be provided. If credentials are provided, secret-agent will use the provided credentials instead.

Credentials file

The credentials are provided to the operator using a kubernetes secret under the GOOGLE_CREDENTIALS_JSON data key. The name of this secret is provided in spec.appConfig.credentialsSecretName. In the default secret-agent deployment, the operator will look for a secret with the provided name in the operator's own namespace. The user can specify a different namespace by setting the argument --cloud-secrets-namespace=[NS_NAME]. If this argument is omitted, the operator's default behavior is to fetch the credentials from the same namespace as the SAC.

The cloud-credentials secret referenced in spec.appConfig.credentialsSecretName would look like this:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: cloud-credentials
  namespace: test-sa
data:
  GOOGLE_CREDENTIALS_JSON: .....[base64 encoded service account json].....

Configure the GCP Secret Manager

Once the necessary credentials are provided to secret-agent using workload identity or a credentials file, the next step is to configure the GCP Secret Manager using the SecretAgentConfiguration.

For example, the following configuration targets GCP Secret Manager for the example-project-id project:

apiVersion: secret-agent.secrets.forgerock.io/v1alpha1
kind: SecretAgentConfiguration
metadata:
  name: standard-forgerock-example
  namespace: test-sa
spec:
  appConfig:
    createKubernetesObjects: true
    credentialsSecretName: cloud-credentials [** skip if using workload identity **]
    secretsManager: GCP
    gcpProjectID: example-project-id

Set up Cloud Backup With Azure Key Vault

note: Azure's API response time on Key Vault is long and will delay the creation of secrets. It might be beneficial to deploy a SAC before long before deploying an application if use Azure Key Vault

The secret-agent uses credentials which are available using two different methods: Azure Managed Identities (recommended for Azure deployemnts) or explicit credentials. Explicit credentials are configured in a secret referenced in the SAC spec spec.appConfig.credentialsSecretName. Example Azure Configuration for a SAC:

apiVersion: secret-agent.secrets.forgerock.io/v1alpha1
kind: SecretAgentConfiguration
metadata:
  name: standard-forgerock-example
  namespace: test-sa
spec:
  appConfig:
    createKubernetesObjects: true
    credentialsSecretName: cloud-credentials [** optional**]
    secretsManager: Azure
    azureVaultName: secret-agent-vault

If no secret is provided in credentialsSecretName, the operator's Azure client will attempt to authenticate using managed identities. For more information, see Azure's documentation. This is the recommended configuration for deployments in Azure's AKS.

Otherwise, the credentials may be explicitly set in the credentialsSecretName secret. The service principle associated with the keys will need the role Key Vault Secrets Officer when using an RBAC policy based Key Vault.

Example credentials secret:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: cloud-credentials
data:
  # AZURE_TENANT_ID: # OPTIONAL: Update if using Azure Key Vault
  # AZURE_CLIENT_ID: # OPTIONAL: Update if using Azure Key Vault
  # AZURE_CLIENT_SECRET: # OPTIONAL: Update if using Azure Key Vault

Note: The maximum secret size supported by Azure is 25Kb For more information, see Azure documentation.

Importing your own secrets

In addition to generating secrets, the secret-agent allow users to import their own secrets. This is especially useful for things like certificates and certificate authorities that can be referenced by other secrets in the SAC. For example, a user can import their organization's CA and use it to sign certificates generated by the secret-agent.

All that is required is to provide a Kubernetes secret with the same name and same key names as described in the SAC. It is important to note that if the cloud backup feature is enabled, the secret to be imported must be provided using the cloud manager's secret manager. The secret-agent will ignore local secrets if cloud backup is enabled.

Naming Convention For Cloud Backups

There is a naming convention used by secret-agent to store and read secrets from the cloud secret managers. In general, the names follow the format:

$prefix-$secretName-$keyName [If secretsManagerPrefix is provided]
$namespace-$secretName-$keyName [If no prefix is provided]

When using a SecretsManagerPrefix writes to a secret manager doesn't use namespace, make sure your prefixes are unique.

Due to cloud provider limitations, all /, . and _ characters in secret names are replaced by - when accessing the cloud secret managers.

For example, consider the following secret agent configuration:

---
apiVersion: secret-agent.secrets.forgerock.io/v1alpha1
kind: SecretAgentConfiguration
metadata:
  name: forgerock-sac
  namespace: dev
spec:
  appConfig:
    secretsManagerPrefix: "devCluster"
    awsRegion: us-east-1
    secretsManager: AWS
  secrets:
  - name: ds-passwords
    keys:
      - name: dirmanager.pw
        type: password

The secret generated by the SAC would be stored as devCluster-ds-passwords-dirmanager-pw in the AWS Secret Manager due to the secretsManagerPrefix. The same name would apply to other cloud providers.

Some key types require more than one secret to be backed up. Such key types require a public and private components stored separately. This is the case for key types: ca, keypair, ssh and keytool. These secrets use the same naming convention described above and append a suffix to the main name as constructed previously.

Key Type	Name
ca	$NAME-pem $NAME-private-pem
keypair	$NAME-pem $NAME-private-pem
ssh	$NAME $NAME-pub
keytool	$NAME $NAME-storepass $NAME-keypass

In the preceding table, $NAME follows the convention at the top of this section.

Examples

We provide a sample SAC that exercises all features of the secret-agent. See the samples folder.

Secret Agent Configuration Schema

The following tables list the configurable parameters of the secret agent configuration (SAC) and their default values.

App Config

Parameter	Description	Default
spec.appConfig.createKubernetesObjects	Create Kubernetes secrets for each generated secret. Can't be set to false if spec.appConfig.secretsManager is set to "none"	true
spec.appConfig.secretTimeout	Set the timeout in seconds for generating each individual secret	40
spec.appConfig.secretsManager	Select the cloud provider to target. If "none", secrets will not be backed up in any cloud secret manager. Can't be set to "none" if spec.appConfig.createKubernetesObjects is false	none
spec.appConfig.secretsManagerPrefix	Prefix added to the name of the secrets stored in the cloud secret manager instead of the namespace.	""
spec.appConfig.credentialsSecretName	Name of the Kubernetes secret containing the credentials to access the cloud provider.	""
spec.appConfig.gcpProjectID	When using GCP as the secret mgr, specify the project ID.	""
spec.appConfig.awsRegion	When using AWS as the secret mgr, specify the region.	""
spec.appConfig.awsKmsKeyId	When using AWS as the secret mgr, you can specifiy the KMS Key Id else will use the default AWS managed KMS key, which poses some limitations on the secret.	""
spec.appConfig.azureVaultName	When using Azure as the secret mgr, specify the vault name.	""
spec.secrets	List of Kubernetes secrets to create. See Secret Config.	[]

Secret Config

Parameter	Description	Default
name	Name of the Kubernetes secret to generate.	""
keys	List of the specs of each key in the Kubernetes secret. See Key Config.	[]

Key Config

Parameter	Description	Default
name	Name of the key in the Kubernetes secret.	""
type	Type of key to generate. Available values: ca;literal;password;ssh;keyPair;truststore;keytool.	""
spec.value	Used when key type is literal. Specify the value of the password.	""
spec.isBase64	Used when key type is literal. If true, interpret the value to be used for the secret as a base64 encoded string.	false
spec.length	Used when key type is password. Specify the length of the password to generate.	32
spec.useBinaryCharacters	Used when key type is password. If true, use the full byte range for each character, not just the ASCII range.	false
spec.algorithm	Used when key type is keyPair. Specify the algorithm used to generate the keyPair.	""
spec.sans	Used when key type is keyPair. Specify alternate DNS names used by the certificate.	""
spec.selfSigned	Used when key type is keyPair. If true, generate a self signed certificate.	false
spec.signedWithPath	Used when key type is keyPair. Specify the path to the CA in the SAC secretname/keyname.	""
spec.duration	Used when key type is is ca or keyPair. Specify the valid duration of the certificate. If a negative duration is specified (-72h) the certificate is generated with an expiry date in the past.	3650d
spec.distinguishedName.country	Used when key type is ca or keyPair. Specify the country name.	""
spec.distinguishedName.organization	Used when key type is ca or keyPair. Specify the organization name.	""
spec.distinguishedName.organizationUnit	Used when key type is ca or keyPair. Specify the organizationUnit name.	""
spec.distinguishedName.locality	Used when key type is ca or keyPair. Specify the locality name.	""
spec.distinguishedName.province	Used when key type is ca or keyPair. Specify the province name.	""
spec.distinguishedName.streetAddress	Used when key type is ca or keyPair. Specify the streetAddress.	""
spec.distinguishedName.postalCode	Used when key type is ca or keyPair. Specify the postalCode.	""
spec.distinguishedName.serialNumber	Used when key type is ca or keyPair. Specify the serialNumber.	""
spec.distinguishedName.commonName	Used when key type is ca or keyPair. Specify the commonName for the certificate.	""
spec.distinguishedName.userId	Used when key type is ca or keyPair. Specify the userId for the certificate.	""
spec.truststoreImportPaths	Used when key type is truststore. List of paths of certificates in the form secretname/keyname that will be imported into the truststore.	""
spec.storeType	Used when key type is keytool. Specify the keystore type. Available values: pkcs12;jceks;jks.	""
spec.storePassPath	Used when key type is keytool. Specify the path to the secret in the SAC to use as the keystore password in the form secretname/keyname.	""
spec.keyPassPath	Used when key type is keytool. Specify the path to the secret in the SAC to use as the key password in the form secretname/keyname.	""
spec.keytoolAliases	Used when key type is keytool. Specify the aliases to include in the keystore. See Keytool Aliases Config.	[]

Keytool Aliases Config

Parameter	Description	Default
name	Name of the alias in the keystore.	""
cmd	keytool command used to create the alias in the keystore. Supported cmds: genkeypair;genseckey;importcert;importpassword;importkeystore.	""
args	Args passed to the keytool command provided in cmd.	""
sourcePath	Used when the keystore cmd is importcert, importpassword or importkeystore. Specify the path to the secret in the SAC to import into the alias in the form secretname/keyname. Note: importcert only imports the public key. importkeystore must be used to import a key pair.	""
isKeyPair	If importing a keypair using importkeystore, must be set to true.	false

Runtime Arguments

Argument	Description	Default
--metrics-addr	The address the metric endpoint binds to. Set to 0 to disable metrics.	":8080"
--health-addr	The address the healthz/readyz endpoint binds to.	":8081"
--enable-leader-election	Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager.	"false"
--cert-dir	Directory where to store/read the webhook certs.	"/tmp/k8s-webhook-server/serving-certs"
--cloud-secrets-namespace	Namespace where the cloud credentials secrets are located. Defaults to the SAC namespace.	SAC's metadata.namespace
--debug	Enable debug logs.	"false"

Running Tests

Tests can be run using make tests. Some of the tests exercise parts of the code that uses keytool, and kubebuildertools. Those must be installed locally in order to run the tests. Another option is to use docker:

• Ensure you're Kubernetes context is pointing to a test cluster, such as minikube, then
  • docker build -t gcr.io/forgerock-io/secret-agent-testing:latest -f --target=tester .
  • docker run -it --rm -v ${PWD}:/root/go/src/github.com/ForgeRock/secret-agent gcr.io/forgerock-io/secret-agent-testing:latest
  • go test ./...