Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cloud configuration support for azure client libraries #1550

Closed
wants to merge 13 commits into from
162 changes: 162 additions & 0 deletions docs/python/clouddiscover.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
# Cloud configuration and discovery

There are multiple Azure Cloud instances. This includes well-known cloud instances such as Public Azure, Azure China Cloud, Azure Government Cloud etc. It also includes private cloud instances that are managed by our customers. An example of this is Azure Stack.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we list everything here? Or are there no major differences to the plan for things like Edge, Hybrid, and Containerized (if what CogSvcs wants to do is different than Hybrid) usage?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to talk to @schaabs about how to possibly plumb auth through from config.


The clouds differ in which endpoints to use when connecting to it. Finding exactly which endpoints to use is challenging for users. This applies both for public clouds and private instances. For private cloud instances, Microsoft does not know the specific endpoints used by the cloud instance. Additionally cloud instances may or may not be reachable from public internet.

In order to discover which endpoints (DNS names and suffixes) are to be used when connecting to a specific cloud, a disovery endpoint has been introduced. This reduces the amount of information a developer needs in order to connect to a given cloud to knowing a single endpoint.
johanste marked this conversation as resolved.
Show resolved Hide resolved

## Goals

* Simplify configuring for a client connecting to public clouds other than Public Azure.
johanste marked this conversation as resolved.
Show resolved Hide resolved
* Allow use of configuration data retreived from cloud configuration discovery endpoint for private cloud instances.
johanste marked this conversation as resolved.
Show resolved Hide resolved

## Non-goals

* Automatically discover which cloud an application runs in.
* Change in requirements for client libraries to always accept a simple name (e.g. storage account name) in addition to full endpoint information when constructing clients.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you anticipate recommending a simple constructor overload using account name (or equivalent) when there is a metadata suffix?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not based on the requirements outlined in this document, no. I don't think sovereign/alternate cloud configurations introduces any new requirement than those we had when the general guidelines to require full endpoints were decided upon.

Do you think it should?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer for this to be an option on client options, not a requirement, though I might be misinterpreting what you mean by "always".

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My intent was to specify that the addition of cloud configurations does not change any requirements to take simple names in addition to full URLs from before. Any service that requires a full URL/endpoint to day will continue to require the user specify a full URL/endpoint.


## Core capabilities

* Well-known (public) cloud configurations MUST be built-in to the client libraries for each language. Each well-known cloud configuration has a "simple name" in addition to a set of key/value pairs representing endpoints and suffixes for known services.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These also contain: (1) authentication endpoints; (2) authentication token audiences for Azure Resource Manager, Graph, and potentially other endpoints in the future. How would you see this impacting authentication libraries and authentication -related http transport policies?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I originally had the auth endpoints per service. But after talking to @schaabs and @chlowell, I moved them to a single "global" section. I'm not entirely happy with the name global, however, so any suggestions on what to rename it to would be very appreciated.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are the keys and what are the values in these KVPs?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally they follow the same pattern and names as what is returned by Azure CLI / PSH, az cloud list.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem with az cloud list is that it is very random in the structure of what it returns.


> The built-in cloud configurations SHOULD be defined in azure core.

> The simple cloud name MAY be a string or an enum value.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why? It seems like some of the requirements below have too much detail. For example, I can imagine cloud names being properties in .NET, e.g. Cloud.Governemt.Enable = true;

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can reduce the requirements if there are other ways to accomplish the same task.

There is a hard requirement to support clouds other than well known clouds. Azure stack is one example. Whatever solution you have would have to support a developer to be able to target two separate stack instances from a single application.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that by introducing the enum version of this we are required to include the clause below about changes being breaking changes.

As much as I like enums, do we want to not do enums so that the config bag can be more flexible and not require a breaking change to change it?


> The cloud configuration instance MAY be represented by a custom type in strongly typed lanugages.
johanste marked this conversation as resolved.
Show resolved Hide resolved

* In addition to the well-known cloud configurations, it MUST be possible to construct a custom cloud configuration instance for a specific cloud (e.g. Azure Stack).

* It MUST be possible (and documented how) to load a configuration from the returned information in ARM's discovery endpoint.

> A canonical use-case for this is to, given the metadata configuration endpoint for a given cloud, download the configuration data to a local file.

### Cloud configuration properties

A cloud configuration object consists of a map of service-name to service-specific-configuration-entry as per below:

```json
{
"<service entry1>": {
"endpoint": "<url>",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we have higher metadata format:

{
   "name": "AzureChina",
   "services": {
        your JSON data
    },
   "name": "MyStack",
   "services": {
        your JSON data
    }
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume that you missed an inner array in the example above. Why do you want to have the "name" attribute? In general, I try to avoid having arrays-with-objects-that-have-a-unique-key. If you don't, you always have to be on the lookout for duplicate items. I'm not sure if having the name of the configuration item in the object itself has enough value to outweigh the uniqueness problem or the fact that the consumer of the configuration (e.g. the clients) can/will do anything with the value.

"suffix": "<suffix>",
"audiences": [
"<optional audiences>"
],
"tenant": "<optional tenant>",
"identityProvider": "<optional identity provider>"
},
"<service entry2>": {
"endpoint": "<url>",
"suffix": "<suffix>",
"audiences": [
"<optional audiences>"
],
"tenant": "<optional tenant>",
"identityProvider": "<optional identity provider>"
}
}
```

where `<service entry>` is (currently) one of the following values:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these new magic strings we're creating? As much as I dislike them, what about using the RP names? https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/azure-services-resource-providers

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it is a 1:1 URL:RP mapping, but I do like the idea of being able to map these via some well known static string like RP name.


|Name|Notes|
|-|-|
|`acr`|Azure Container Registry|
|`authentication`||
|`batch`|Batch service endpoint|
|`dataLakeCatalogAndJob`||
|`dataLakeFileSystem`||
|`microsoftGraph`||
|`keyvault`||
|`media`|Media services service endpoint|
|`portal`|Portal address|
|`resourceManager`||
|`sqlServer`||
|`storage`||

None of the properties in a service configuration entry are required as they differ between different services.

> Note that this format differs from the raw json exposed by the discovery endpoint. The endpoint discovery response is not structured on a per-service basis, which makes it harder to evolve the set of metadata in the response.

> A given language can choose to expose the configuration as a dictionary or as a strongly typed class.

## Service specific client library guidance

* Service specific client libraries that have baked-in cloud-specific defaults MUST accept a cloud configuration instance in the constructor (or equivalent) for the service client instance. Client libraries MAY also accept a a simple cloud name.
johanste marked this conversation as resolved.
Show resolved Hide resolved

### Ambient default configuration

* A language MAY provide the capability to set the default cloud to use.
* If supported, client libraries MUST copy the applicable configuration settings when creating the client (or other configuration-aware) object. If the default configuration is changed after the object has been created, the change MUST NOT be observed by the already created object.

> In most cases, the ambient default configuration will be set during application start-up, much like other process wide configuration data is initialized (e.g. log levels etc.)

### Interaction with other configuration settings

The order of precedence is (in order of decreasing specificity):

* Explicitly provided endpoint parameter passed in value in the method call
* Explicitly provided configuration value from cloud_configuration parameter passed in the method call
* Ambient default cloud (if supported)
johanste marked this conversation as resolved.
Show resolved Hide resolved
* Endpoint specific environment variables (e.g. `AZURE_AUTHORITY_HOST`)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to list the env variables that are mapping your JSON above? In your table above?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If not the specific env variable perhaps we can provide a well known pattern for property to env variable name so that we can start to have consistency in how they env variables are named.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a list of "well known" environment variables in the general design guidelines. Regarding naming, it only includes the requirement that they are prefixed with AZURE_. Should we have more specific guidance?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if there was more guidance around how those look so that we have consistency but I suppose we have to wait and see what patterns emerge to create such a guideline.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just an observation; we had a flurry of variables getting added early on, but the list hasn't grown much since then.

jongio marked this conversation as resolved.
Show resolved Hide resolved
* Final fallback, corresponding Public Azure's settings (built in to the client library)
JonathanGiles marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be reasonable to start with simple chain? client setting > global setting > default

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is more or less what I have here - with the only twist that we already have bespoke environment variables for some settings (e.g. AZURE_AUTHORITY_HOST)


### Missing configuration properties

Given that more configuration settings can be added over time, you may run into a situation where not all configuration settings that a client library needs are populated in a provided cloud configuration entry. If a cloud configuration is provided, but a client library cannot find the configuration setting it expects, the method MUST fail.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be great to have a fall back mechanism.

i.e., if we don't find the value, then use the user supplied default.

var endpoint = getendpoint("KEY", "default endpoint")


> In most languages, the failure is exposed as an exception.

* Methods that accept a configuration object that is missing one or more configuration settings that it expects MUST raise an error. The error message MUST be clear enough for the developer to understand what data is missing and how to supply it.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How are we going to develop it and test it? Seems like it would be very expensive. Maybe LLC is the answer here, i..e when call fails, the users needs to modify the request.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nah, look in the configuration passed in and raise an exception if something you needed isn't available.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you write one or two good sample error messages, everyone will copy them and they'll be consistently good.


* Adding a new required configuration property for a given method/API is a breaking change.
Comment on lines +136 to +138
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm having trouble on these two points. I read above that configuration was passed in at construction of the service client. I don't know how these two points above relate to that - in what context is there a method that accepts a configuration object, or are you talking about validating the configuration at service client instantiation time?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this context, the constructor is a "method" (or it can be a factory method). Or, for languages that support module level methods etc (e.g. Python), a module level helper methods. In other words, in practical terms, this is most commonly going to apply to construction of clients, but it also can apply to anything that, internally, creates a service client instance.


## Example usage

### Python
```python
# In order to default to switch all default values to match the expected configuration for
# the Azure China Cloud.
azure.core.settings.set_default_cloud('AzureChinaCloud')

creds = azure.identity.DefaultAzureCredential() # We will use the Azure China Cloud's authority (https://login.chinacloudapi.cn) since that is what is configured as the default cloud.

# The default management endpoint from AzureChinaCloud is used
# (picked up from the resourceManager service entry in of the AzureChinaCloud built-in cloud configuration)
client = azure.mgmt.compute.ComputeManagementClient(subscriptionId, creds)

# I can still specify the default authority host to override the default settings...
public_azure_creds = azure.identity.DefaultAzureCredential(authority='https://login.windows.net')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm assuming I could also pull that magic string from azure.core.clouds.public?


# I can also specify a full cloud configuration object:
# In order to default to switch all default values to match the expected configuration for
# the Azure China Cloud.
config = CloudConfiguration.from_json(requests.get('https://azurestackinstance1.contoso.com/discover'.json()))
johanste marked this conversation as resolved.
Show resolved Hide resolved
johanste marked this conversation as resolved.
Show resolved Hide resolved
azure.core.settings.set_default_cloud(config['AzureStack'])

# ...or pass in a custom cloud configuration into a method
creds = azure.identity.DefaultAzureCredential(cloud_configuration=config['PublicAzure'])
```

### C#

```C#
// In order to default to switch all default values to match the expected configuration for
// the Azure China Cloud.
creds = Azure.Core.Identity.DefaultAzureCredentials(
Azure.Core.Identity.DefaultAzureCredentialOptions() {
CloudConfiguration = Azure.Core.CloudConfigurations.AzureChinaCloud
}
);
```
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not clear to me why we offer so many options. Is there a reason besides that we can?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a common practice in Python to "overload" methods that take "well-known instance with name in addition to custom instances" to, accept the name.


Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to very roughly translate that to a first guess at how I could imagine C# looking:

using Azure.Core;
using Azure.Identity;
using Azure.ResourceManager.Compute;

// In order to default to switch all default values to match the expected configuration for
// the Azure China Cloud.
AzureConfiguration.Default = AzureConfiguration.AzureChinaCloud; // or AzureConfiguration["AzureChinaCloud"];

// We will use the Azure China Cloud's authority (https://login.chinacloudapi.cn) since that is what is configured as the default cloud.
var creds = new DefaultAzureCredential();

// The default management endpoint from AzureChinaCloud is used
// (picked up from the resourceManager service entry in of the AzureChinaCloud built-in cloud configuration)
var client = new ComputeManagementClient(subscriptionId, creds)

// I can still specify the default authority host to override the default settings...
var options = new DefaultAzureCredentialOptions { AuthorityHost = new Uri("https://login.windows.net") };
var publicAzureCreds = new DefaultAzureCredential(options);

// I can also specify a full cloud configuration object:
// In order to default to switch all default values to match the expected configuration for
// the Azure China Cloud.
AzureConfiguration.Default = AzureConfiguration.AzureChinaCloud; // or AzureConfiguration["AzureChinaCloud"];

// ...or pass in a custom cloud configuration instance into a method
var config = await AzureConfiguration.GetAsync(new Uri("https://azurestackinstance1.contoso.com/discover?api-version=2019-05-01"));
options = new DefaultAzureCredentialOptions { AzureConfiguration = config["AzureStackInstance1"] };
creds = new DefaultAzureCredential(options);

// ...or by name
options = new DefaultAzureCredentialOptions { AzureConfiguration = AzureConfiguration.AzurePublicCloud };
creds = new DefaultAzureCredential(options);

## Future evolution/addition of per-service endpoint configuration

* Over time metadata can be added for a given service/new services may be added. Client libraries that depend on the new metadata can set the minimum version of the azure core dependency to ensure that the new metadata is available for well-known clouds.
johanste marked this conversation as resolved.
Show resolved Hide resolved

## Issues

* Only updated client libraries will pick up the ambient default settings. This problem will fade over time as new client library versions are made aware of ambient settings.