Create a Azure KeyVault based configuration provider

[muratg]

CoreCLR based KeyVault SDK will be ready late August per @glennc.

Let's sit down and come up with the design.

[brentschmaltz]

@muratg @glennc how is this progressing? looks interesting.

[glennc]

@brentschmaltz We haven't started any real work yet, just some initial discussion and design.

[cwe1ss]

Given two vaults:

• myvault1-europe - https://myvault1-europe.vault.azure.net
  • Sql-ConnectionString: Server=...
  • Sql-ConnectionString-Module2: Server=...
  • ApplicationInsightsKey: 11111111-2222-3333-4444-555555555555
  • SomePassword: 123456
• myvault2-us - https://myvault2-us.vault.azure.net
  • DbConnectionString: Server=...
  • ApplicationInsightsKey: 11111111-2222-3333-4444-555555555555
  • SomeImportantKey: SecretValue

I'd like to do the following things:

• I want to reference the vault by alias in my code - e.g. config["myvault1"] instead config["myvault1-europe"].
  • I do NOT want to use Configuration["myvault1-europe:..."] because this name will likely be different for every environment (dev, production)
  • I could also have to change it for failover/scale out reasons.
• I want to reference secret names by alias - e.g config["myvault1:ConnectionString"] instead config["myvault1:Sql-ConnectionString"].
  • Key Vaults may be managed by different people who have their own naming structure.
  • Secret names could contain characters which are not valid C# characters for properties and therefore break mapping to objects.
• I want to use multiple aliases for a single secret - e.g. config["myvault1:CnnStr"] and config["myvault1:SqlConnectionString"] where both return myvault1-europe/Sql-ConnectionString
  • There could be different Options-classes with different property names - e.g. MyModule1Options.CnnStr, MyModule2Options.SqlConnectionString.
• The secret aliases should support a way to map multiple physical secrets to Options-classes with the same property name
  • Example: I have Module1Options.ConnectionString which should point to Sql-ConnectionString and Module2Options.ConnectionString which should point to Sql-ConnectionString-Module2
• I want to specify that all secrets from a vault should be loaded
• I want to specify that only a given set of secrets should be loaded
  • The system shouldn't have secrets in memory which it doesn't need
• I want to be able to use multiple vaults
• I want to be able to decide whether both vaults are loaded into a single IConfiguration or multiple ones
• There should be an option to throw an exception if the specified secret alias is not found in the vault
  • This should make sure typos etc. are noticed
• I want to be able to specify the clientId and clientSecret/clientThumbprint which is required to access the vault from another IConfiguration element (to get it from environment variables or a different file)

To support most of this, we ended up building a solution that uses a two step process for loading secrets - we have a config file which defines the mappings and this is then used to load the secrets. This is how we use it:

KeyVaults.json - this defines the aliases and is used in alle environments:

{
  "KeyVaults": {
    "myvault1": {
      "Secrets": {
        "DbConnectionString": "Sql-ConnectionString",
        "Module1/SqlConnectionString": "Sql-ConnectionString",
        "Module2/SqlConnectionString": "Sql-ConnectionString-Module2",
        "ApplicationInsightsKey": "ApplicationInsightsKey"
      }
    }
  }
}

KeyVaults.Development.json - this defines the physical names of the vaults and the access credentials:

{
  "ClientID": "75e92952-701b-43f7-9f8b-8769bdd804de",
  "CertThumbprint": "a909502dd82ae41433e6f83886b00d4277a32a7b",
  "StoreLocation": "LocalMachine",
  "KeyVaults": {
    "myvault1": {
      "KeyVaultName": "myvault1-europe"
    }
  }
}

On startup, we first load this configuration with a ConfigurationBuilder and then we have a separate IKeyVaultProvider which is used to access the secrets. We also have a helper method which allows us to map stuff from this provider into an object using IServiceCollection.Configure<Options>. So, in code we can use it like this: services.Configure<Module1Options>(KeyVaultProvider, "myvault1", prefix: "Module1");

I could provide some code if you're interested - it's not really beautiful though 😄

[glennc]

Is the selection of europe vs us known at startup and doesn't change? or is it that some code paths want to use one and some want to use the other?

[cwe1ss]

Our current scenario is that we have different vaults for different environments. this means, it is known at startup and doesn't change.

If we would scale-out one service to additional data centers we might just use another environment key and create another mapping file. So this case would also be known at startup. (We don't yet have this scenario so I haven't given much thought into it)

We also don't yet regularly refresh secrets, so Key Vault outages while our application is running are no concern for us atm.

Not sure what requirements people have that use multi-tenancy etc.

[glennc]

Ok cool, so connecting to different key vault instances in different environments is fine then, with some code when you add it. However, the ability to ask for "ConnStr" and have that map to two different named keys in the vault isn't something that I had thought about at all.

You want this because you don't control the vaults and they end up with slightly different names. Right?

[cwe1ss]

exactly! let's say we have one big Key Vault for our application that contains connection strings for all modules (each uses a separate DB). In Key Vault, you would have to give them different names - e.g. "CustomersConnectionString", "SalesConnectionString", ...

In code, you might have separate libraries for each module and each might have an options class - e.g. CustomersOptions, which just contains a ConnectionString property.

If you want to use services.Configure<CustomerOptions>(...) there must be some way to map the secret "CustomersConnectionString" to the property "ConnectionString".

[cwe1ss]

of course, all of this might be pretty advanced - maybe having just a 1:1 mapping covers most people?! but for us, this was/is essential.

[glennc]

Is your app ASP.NET Core or are you dumping all the things you needed for a different app?