Determine plan for managing secrets

[matthchr]

Once an ARM service has been created there are some which have keys or secrets associated with them.

For example, a storage account has storage keys, a Redis account has keys, etc. These keys need to be stored and access has to be given to the right users.

ASO historically had two approaches for this, originally storing secrets using Kubernetes secrets, and now optionally supports storing secrets in a KV instead (value for this is controlled via env variable):
AZURE_OPERATOR_KEYVAULT

Take a look at ASO rediscache_types.go for example:

	SecretName             string               `json:"secretName,omitempty"`
	KeyVaultToStoreSecrets string               `json:"keyVaultToStoreSecrets,omitempty"`

[frodopwns]

ASO lets some services specify the Key Vault its credentials should be placed in. Also there is a global setting that lets all credentials be placed in a specified Key Vault.

[matthchr]

We probably need a holistic secrets/identity design - see the following for other secrets related issues or inconsistencies:

1. Feature: Secrets created by ASO should have configurable annotations #1398
2. Database users should be intelligent about which identity to use when provisioning users #1353
3. The path to the secret for a specific resource should be in the status #1318
4. Feature: Secret deletion should be a required part of resouce deletion #1280
5. Add password field to VirtualMachine #1167

Some customers also want the ability to do a sort of a-b deployment where they swap resources between namespaces periodically (turning off reconciling in one). This isn't a problem for pure resources associated with no secrets, but for resources associated with secrets questions end up cropping up about how to get the secret from one namespace to another (since ASO created it)

I think we want to get out of the world where we generate credentials or secrets in the users behalf. It makes some rollover scenarios harder and means we need to be really sure we're generating cryptographically secure credentials. Inevitably some organization will have a more stringent key-length or password strength requirement than we're complying to. Instead we might want to investigate:

For places where we generate usernames and/or passwords today (example: MySQL, AzureSQL, etc): Put the username in the spec, and the password as a linked Kubernetes (optionally KV?) secret.

For places where Azure generates the secret and returns it only on create (example: Storage): Continue doing what we're doing? To make this work with "swap" scenarios we may need to have some logic to go get the keys again if we can't find them.

For places where Azure generates the secret one time and it cannot be retrieved later: Unsure - we can do the same as we're doing now but it doesn't easily support migration between namespace scenarios because the new namespace won't have the secret. We may need to do some thinking on this.

[matthchr]

A number of resources we're already code generating have various kinds of secrets. These secrets need to be exposed in a secure fashion that also allows the resources to be used.

Here are a few examples:

1. StorageAccount has keys accessible through a ListKeys API (which we don't currently make).
2. ServiceBusNamespace has a connection string, which seems accessible through a child resource that is automatically created under the ServiceBusNamespace when it's created. Accessible via: az servicebus namespace authorization-rule keys list -n RootManageSharedAccessKey -g <group> --namespace-name <namespace> --query primaryConnectionString -o tsv (see this StackOverflow question
3. VirtualMachineScaleSets have a username and password which must be provided to the API - this is currently just part of the spec but probably shouldn't be (for security reasons)
4. CosmosDB supports a variety of authentication schemes (see secure access to data). There's a ListKeys and ListReadOnlyKeys API, but also supports RBAC with AAD identities.

[matthchr]

Worth looking into https://akv2k8s.io/ and secret store CSI driver

[vidaks]

I have been experimenting with ASO v2 and quickly ran into this issue.

While shared keys proabably has its audience, I believe it´s mostly from the lack
of alternatives..

And I don´t agree with the premise that shared keys for accessing Azure
resources is the way to go.

Keys can leak, they must be rotated regularly and they incur maintenance costs
and security concerns.

A more preferable method would in my opinion be the use of managed
identities and authentication with OAuth2 for access to all Azure resources.
And being able to manage all resources, including identities and role
assignments, using the same management tool (ASO).

In summary being able to do the following via ASO ():

• Create a managed identity
• Create a resource in Azure (e.g Storage Account)
• Assign the managed identity a role on the given resource
• Bind the managed identity in Azure to a specific AKS service account

Basically using Pod Identity in a fully managed way to enable developers to
manage all resources they need for their applications using one management tool.

I suppose this can be consider this a feature request :)

[matthchr]

@vidaks - Yes we agree. Using managed identity is overall much cleaner for a lot of scenarios. We still will need to figure out secrets support for cases where for whatever reason customers cannot use managed identity, but managed identity should probably be the default, or at least the heavily preferred option.

[matthchr]

#1828 is tracking the required managed identity support