- Author: Nick Beenham (@superbeeny)
Currently the only way to provide kubernetes secrets to a container is to mount them through a secret volume. This is not ideal for many use cases, especially when the secret is only needed in the environment variables of the container. This proposal aims to add support for secret stores to the environment variables of a container.
Kubernetes secret: A Kubernetes object that stores sensitive data, such as passwords, OAuth tokens, and ssh keys. Putting this information in a secret is safer and more flexible than putting it verbatim in a pod definition or in a docker image. See the documentation here.
Issue Reference: Issue #5520
- Allow users to provide Kubernetes secrets to a container through environment variables
- (out-of-scope): Integration with other secret stores besides Kubernetes. This is tracked by other issues.
- (out-of-scope): Other users for secrets besides environment variables. This is tracked by other issues.
As a radius user, I want to provide a secret to a container through an environment variable so that I can avoid mounting the secret as a volume. I want to be able to reference the secret within the application bicep file.
extension radius
@description('The Radius Application ID. Injected automatically by the rad CLI.')
param application string
resource demo 'Applications.Core/containers@2023-10-01-preview' = {
name: 'demo'
properties: {
application: application
container: {
image: 'ghcr.io/radius-project/samples/demo:latest'
ports: {
web: {
containerPort: 3000
}
}
env: {
DB_USER: { value: 'DB_USER' }
DB_PASSWORD: {
valueFrom: {
secretRef: {
source: secret.id
key: 'DB_PASSWORD'
}
}
}
}
}
}
}
resource secret 'Applications.Core/secretStores@2023-10-01-preview' = {
name: 'secret'
properties: {
application: application
data: {
DB_PASSWORD: {
value: 'password'
}
}
}
}
To reference a secret directly:
env: {
DB_USER: { value: 'DB_USER' }
DB_PASSWORD: {
valueFrom: {
secretRef: {
source: 'myKubernetesSecret'
}
}
}
}The design of this new feature will require updates to the versioned datamodel, the conversion functions, the containers typespec and the common typespec. These will be breaking changes to the schema. Users will need to update the environment variables in their bicep files to use the new secret reference type.
The design of this feature will require updates to the versioned datamodel, the conversion functions, the containers typespec and the common typespec to leverage the new secret reference type and provide support for secret stores in environment variables beyond the current support for environment variables with a string value.
Advantages of this approach are that it allows users to provide secrets to a container through environment variables. This is a common use case and will make it easier for users to provide secrets to their containers. In using much of the existing functionality of Radius, this approach is also relatively simple to implement.
Disadvantages are that it will break existing bicep files that use environment variables. Users will need to update their bicep files to use the new secret reference type.
We first convert the versioned datamodel to a base datamodel that can handle secrets.
// toEnvDataModel: Converts from versioned datamodel to base datamodel
func toEnvDataModel(e map[string]*EnvironmentVariable) (map[string]datamodel.EnvironmentVariable, error) {
m := map[string]datamodel.EnvironmentVariable{}
for key, val := range e {
if val == nil {
return nil, v1.NewClientErrInvalidRequest(fmt.Sprintf("Environment variable %s is nil", key))
}
if val.Value != nil && val.ValueFrom != nil {
return nil, v1.NewClientErrInvalidRequest(fmt.Sprintf("Environment variable %s has both value and secret value", key))
}
if val.Value != nil {
m[key] = datamodel.EnvironmentVariable{
Value: val.Value,
}
} else {
m[key] = datamodel.EnvironmentVariable{
ValueFrom: &datamodel.EnvironmentVariableReference{
SecretRef: &datamodel.EnvironmentVariableSecretReference{
Source: to.String(val.ValueFrom.SecretRef.Source),
Key: to.String(val.ValueFrom.SecretRef.Key),
},
},
}
}
}
return m, nil
}
// fromEnvDataModel: Converts from base datamodel to versioned datamodel
func fromEnvDataModel(e map[string]datamodel.EnvironmentVariable) map[string]*EnvironmentVariable {
m := map[string]*EnvironmentVariable{}
for key, val := range e {
if val.Value != nil {
m[key] = &EnvironmentVariable{
Value: val.Value,
}
} else {
m[key] = &EnvironmentVariable{
ValueFrom: &EnvironmentVariableReference{
SecretRef: &EnvironmentVariableSecretReference{
Source: to.Ptr(val.ValueFrom.SecretRef.Source),
Key: to.Ptr(val.ValueFrom.SecretRef.Key),
},
},
}
}
}
return m
}
Updates to the container typespec to allow for secret references in environment variables. This replaces the existing environment variable type of map[string]string to allow for a secret reference.
containers.tsp
@doc("Definition of a container")
model Container {
@doc("The registry and image to download and run in your container")
image: string;
@doc("The pull policy for the container image")
imagePullPolicy?: ImagePullPolicy;
@doc("environment")
env?: Record<EnvironmentVariable>;
@doc("container ports")
ports?: Record<ContainerPortProperties>;
@doc("readiness probe properties")
readinessProbe?: HealthProbeProperties;
@doc("liveness probe properties")
livenessProbe?: HealthProbeProperties;
@doc("container volumes")
volumes?: Record<Volume>;
@doc("Entrypoint array. Overrides the container image's ENTRYPOINT")
command?: string[];
@doc("Arguments to the entrypoint. Overrides the container image's CMD")
args?: string[];
@doc("Working directory for the container")
workingDir?: string;
}
@doc("Environment variables type")
model EnvironmentVariable {
@doc("The value of the environment variable")
value?: string;
@doc("The reference to the variable")
valueFrom?: EnvironmentVariableReference;
}
@doc("The reference to the variable")
model EnvironmentVariableReference {
@doc("The secret reference")
secretRef: SecretReference;
}
We also need to move the SecretReference type to the common typespec so that it can be used in multiple places.
common.tsp
@doc("This specifies a reference to a secret. Secrets are encrypted, often have fine-grained access control, auditing and are recommended to be used to hold sensitive data.")
model SecretReference {
@doc("The ID of an Applications.Core/SecretStore resource containing sensitive data required for recipe execution.")
source: string;
@doc("The key for the secret in the secret store.")
key: string;
}The renderer will need to be updated in several areas to handle the new secrets implementation.
The function GetDependencyIDs will need to be updated to handle the new secret reference type. This function will need to determine if the environment variable is a secret reference or a string. The function will also need to determine whether the secret is a radius resource or a Kubernetes secret.
The function convertEnvVar will need to be created to facilitate the conversion of map[string]EnvironmentVariable to map[string]corev1.EnvVar. The function will need to handle resolving the secret coming from a Kubernetes secret or a Radius resource ID.
Error handling is covered within the functions and Radius errors are used where appropriate.
- The addition of tests for the conversion functions from and to the versioned datamodel to the base datamodel.
- The addition of tests for the conversion functions from the base datamodel to the versioned datamodel.
- Add tests to test for errors and negative test cases.
- Add tests to ensure that secrets can be referenced in environment variables.
The handling of secrets will remain within Kubernetes and Radius is only providing a way to reference these secrets in the environment variables of a container. This is an improvement over the current method of mounting secrets as volumes as it allows for more flexibility and security. Also, the secrets are stored in the Kubernetes secret store and are never exposed to the user.
These will be breaking changes to the schema. Users will need to update the environment variables in their bicep files to use the new secret reference type.
No additional monitoring or logging is required for this feature.
Work completed in a pair programming session with a second developer. The work will be broken down into the following tasks:
- Update the versioned datamodel to include the new secret reference type
- Update the conversion functions to handle the new secret reference type
- Update the containers typespec to include the new secret reference type
- Update the common typespec to include the new secret reference type
- Update the functional tests to cover the new functionality
- Update the documentation to include the new functionality
The terraform resource provider also implemented a similar feature to this. There was a difference in design that needed to be resolved to maintain a consistent user experience across the two resource providers.
The two options were discussed and it was decided that this implementation would be adopted for the following reasons.
- This solution is more consistent with existing Kubernetes design patterns and is more user-friendly.
- Secrets and environment variables are closely related in Kubernetes and it makes sense to allow users to reference secrets in environment variables. This design also allows for other types of references in the future.