Automate team subscription configuration keyvault provisioning

[benbp]

We have a way to add custom live test configuration overlays that can be managed directly by partner teams (see here and here). This process has been ad-hoc thus far as it was not common. It is both poorly documented and consists of manual steps.

More and more teams are starting to request this functionality, so it should be automated. This could either take the form of automating the ad-hoc steps via a tool/script OR adding some sort of configuration alongside the test yaml that could hint our pipeline generator to do it instead.

Provisioning steps today:

1. Provision a new keyvault
   • Update keyvault owners and access policies for partner team
   • Add new subscription configuration secret(s) to the keyvault
2. Create a new devops variable group backed by the keyvault.
   • This requires initializing a new service connection to access this keyvault and giving it the appropriate access policies. Only a few admins can do this since the keyvaults are stored in our main live testing secrets resource group.
3. Make changes to add reference to custom config, add a new cloud definition, and remove any hardcoded keyvault variable references. For example: Update text analytics live test config to support sovereign cloud testing azure-sdk-for-python#22376
4. Edit the live and weekly test pipelines to add the variable group for that Keyvault. Click “Edit”, then click the three dots and “Triggers”. Click the “Variables” tab then “Link Variable Group”. Search for the live test secrets variable group and add it. Then click Save.
5. Run the pipeline. It will give you a one-time prompt to authorize the pipeline to use the service connection for that variable group’s backing Keyvault.
6. If the subscription configuration references a custom subscription separate from our main testing subscriptions, add the subscription configuration to the live-test-cleanup pipeline for zombie resource cleanup.

[benbp]

FYI @heaths @weshaggard I'll still probably manually spin up the cognitive language tests group, but I think after that one this will be the fourth or fifth time I've done this, so I think the automation effort is worth the investment now.

[heaths]

Be sure to also grant some sort of reader-type role to the service connection. I had to have @weshaggard grant me some sort of access in order to actually read the Key Vault secrets when adding a variable to the variable group. I had some sort of admin / write access to the KV, RG, and VG all, but when you click + Add on the variable group, I got an error about insufficient privileges on the service connection to the KV.

[heaths]

I also mentioned one other idea to @weshaggard but it may not be practical given ambiguous times to sync between an Azure AD and DevOps: create an Azure AD group into which partner team "admins" are added then create a DevOps team based on that to grant privileges to the service connection and VG. In theory it would be a good way to manage admins for that pipeline, but in practice the delay would likely make it difficult.

[benbp]

I am also considering consolidating service connections or otherwise having a simpler way of accessing the keyvaults. I don't think we necessarily need to be spinning up a new SP per keyvault. However, perhaps that will make it harder for other users to edit the variable group if they are adding new secrets to the keyvault? Since I'm admin on all of them I don't necessarily see the permission limited view that you have.

[heaths]

Adding new secrets, IMO, should be a P1. P2 at worst. As new tests are added - especially for services like Cognitive that might themselves connected to many other services - new secrets for connection strings, etc. need to be added. I could almost self-service if not for the service connection issue.