From 777653b41e58cd298f4d4da25e8476e0a4c5b22d Mon Sep 17 00:00:00 2001 From: Johan Stenberg Date: Mon, 13 Jul 2020 12:33:29 -0700 Subject: [PATCH 01/12] Initial revision, clouddiscover.md --- docs/python/clouddiscover.md | 156 +++++++++++++++++++++++++++++++++++ 1 file changed, 156 insertions(+) create mode 100644 docs/python/clouddiscover.md diff --git a/docs/python/clouddiscover.md b/docs/python/clouddiscover.md new file mode 100644 index 00000000000..9442f1d73ba --- /dev/null +++ b/docs/python/clouddiscover.md @@ -0,0 +1,156 @@ +# Cloud configuration and discovery + +There are multiple Azure Cloud instances. This includes well-known cloud instances such as Public Azure, Azure China Cloud, Azure Government Cloud etc. It also includes private cloud instances that are managed by our customers. An example of this is Azure Stack. + +The clouds differ in which endpoints to use when connecting to it. Finding exactly which endpoints to use is challenging for users. This applies both for public clouds and private instances. For private cloud instances, Microsoft does not know the specific endpoints used by the cloud instance. Additionally cloud instances may or may not be reachable from public internet. + +In order to discover which endpoints (DNS names and suffixes) are to be used when connecting to a specific cloud, a disovery endpoint has been introduced. This reduces the amount of information a developer needs in order to connect to a given cloud to knowing a single endpoint. + +## Goals + +* Simplify configuring for a client connecting to public clouds other than Public Azure. +* Allow use of configuration data retreived from cloud configuration discovery endpoint for private cloud instances. + +## Non-goals + +* Automatically discover which cloud an application runs in. +* Change in requirements for client libraries to always accept a simple name (e.g. storage account name) in addition to full endpoint information when constructing clients. + +## Core capabilities + +* Well-known (public) cloud configurations MUST be built-in to the client libraries for each language. Each well-known cloud configuration has a "simple name" in addition to a set of key/value pairs representing endpoints and suffixes for known services. + +> The built-in cloud configurations SHOULD be defined in azure core. + +> The simple cloud name MAY be a string or an enum value. + +> The cloud configuration instance MAY be represented by a custom type in strongly typed lanugages. + +* In addition to the well-known cloud configurations, it MUST be possible to construct a custom cloud configuration instance for a specific cloud (e.g. Azure Stack). + +* It MUST be possible (and documented how) to load a configuration from the returned information in ARM's discovery endpoint. + +> A canonical use-case for this is to, given the metadata configuration endpoint for a given cloud, download the configuration data to a local file. + +### Cloud configuration properties + +A cloud configuration object consists of a map of service-name to service-specific-configuration-entry as per below: + +```json +{ + "": { + "endpoint": "", + "suffix": "", + "audiences": [ + "" + ], + "tenant": "", + "identityProvider": "" + }, + "": { + "endpoint": "", + "suffix": "", + "audiences": [ + "" + ], + "tenant": "", + "identityProvider": "" + } +} +``` + +where `` is (currently) one of the following values: + +|Name|Notes| +|-|-| +|`acr`|Azure Container Registry| +|`authentication`|| +|`batch`|Batch service endpoint| +|`dataLakeCatalogAndJob`|| +|`dataLakeFileSystem`|| +|`microsoftGraph`|| +|`keyvault`|| +|`media`|Media services service endpoint| +|`portal`|Portal address| +|`resourceManager`|| +|`sqlServer`|| +|`storage`|| + +None of the properties in a service configuration entry are required as they differ between different services. + +> Note that this format differs from the raw json exposed by the discovery endpoint. The endpoint discovery response is not structured on a per-service basis, which makes it harder to evolve the set of metadata in the response. + +> A given language can choose to expose the configuration as a dictionary or as a strongly typed class. + +## Service specific client library guidance + +* Service specific client libraries that have baked-in cloud-specific defaults MUST accept a cloud configuration instance in the constructor (or equivalent) for the service client instance. Client libraries MAY also accept a a simple cloud name. + +### Ambient default configuration + +* A language MAY provide the capability to set the default cloud to use. +* If supported, client libraries MUST copy the applicable configuration settings when creating the client (or other configuration-aware) object. If the default configuration is changed after the object has been created, the change MUST NOT be observed by the already created object. + +> In most cases, the ambient default configuration will be set during application start-up, much like other process wide configuration data is initialized (e.g. log levels etc.) + +### Interaction with other configuration settings + +The order of precedence is (in order of decreasing specificity): + +* Explicitly provided endpoint parameter passed in value in the method call +* Explicitly provided configuration value from cloud_configuration parameter passed in the method call +* Ambient default cloud (if supported) +* Endpoint specific environment variables (e.g. `AZURE_AUTHORITY_HOST`) +* Final fallback, corresponding Public Azure's settings (built in to the client library) + +### Missing configuration properties + +Given that more configuration settings can be added over time, you may run into a situation where not all configuration settings that a client library needs are populated in a provided cloud configuration entry. If a cloud configuration is provided, but a client library cannot find the configuration setting it expects, the method MUST fail. + +> In most languages, the failure is exposed as an exception. + +* Methods that accept a configuration object that is missing one or more configuration settings that it expects MUST raise an error. + +## Example usage + +### Python +```python +# In order to default to switch all default values to match the expected configuration for +# the Azure China Cloud. +azure.core.settings.set_default_cloud('AzureChinaCloud') + +creds = azure.identity.DefaultAzureCredential() # We will use the Azure China Cloud's authority (https://login.chinacloudapi.cn) since that is what is configured as the default cloud. + +# I can still specify the default authority host to override the default settings... +public_azure_creds = azure.identity.DefaultAzureCredential(authority='https://login.windows.net') + +# I can also specify a full cloud configuration object: +# In order to default to switch all default values to match the expected configuration for +# the Azure China Cloud. +config = CloudConfiguration.from_json(requests.get('https://azurestackinstance1.contoso.com/discover'.json())) +azure.core.settings.set_default_cloud(config['AzureStack']) + +# ...or pass in a custom cloud configuration into a method +creds = azure.identity.DefaultAzureCredential(cloud_configuration=config['PublicAzure']) +``` + +### C# + +```C# +// In order to default to switch all default values to match the expected configuration for +// the Azure China Cloud. +creds = Azure.Core.Identity.DefaultAzureCredentials( + Azure.Core.Identity.DefaultAzureCredentialOptions() { + CloudConfiguration = Azure.Core.CloudConfigurations.AzureChinaCloud + } +); +``` + +## Future evolution/addition of per-service endpoint configuration + +* Over time metadata can be added for a given service/new services may be added. Client libraries that depend on the new metadata can set the minimum version of the azure core dependency to ensure that the new metadata is available for well-known clouds. + +## Issues + +* Only updated client libraries will pick up the ambient default settings. This problem will fade over time as new client library versions are made aware of ambient settings. + From c16a4094fd0ca7ee599a8c5e1f9e0f10faaeab9c Mon Sep 17 00:00:00 2001 From: Johan Stenberg Date: Wed, 29 Jul 2020 13:58:55 -0700 Subject: [PATCH 02/12] Added some minor clarifications --- docs/python/clouddiscover.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/python/clouddiscover.md b/docs/python/clouddiscover.md index 9442f1d73ba..5aecd7067eb 100644 --- a/docs/python/clouddiscover.md +++ b/docs/python/clouddiscover.md @@ -109,7 +109,9 @@ Given that more configuration settings can be added over time, you may run into > In most languages, the failure is exposed as an exception. -* Methods that accept a configuration object that is missing one or more configuration settings that it expects MUST raise an error. +* Methods that accept a configuration object that is missing one or more configuration settings that it expects MUST raise an error. The error message MUST be clear enough for the developer to understand what data is missing and how to supply it. + +* Adding a new required configuration property for a given method/API is a breaking change. ## Example usage @@ -121,6 +123,10 @@ azure.core.settings.set_default_cloud('AzureChinaCloud') creds = azure.identity.DefaultAzureCredential() # We will use the Azure China Cloud's authority (https://login.chinacloudapi.cn) since that is what is configured as the default cloud. +# The default management endpoint from AzureChinaCloud is used +# (picked up from the resourceManager service entry in of the AzureChinaCloud built-in cloud configuration) +client = azure.mgmt.compute.ComputeManagementClient(subscriptionId, creds) + # I can still specify the default authority host to override the default settings... public_azure_creds = azure.identity.DefaultAzureCredential(authority='https://login.windows.net') From 1bad913d5a5bae84cf03bc9a801514d996802c64 Mon Sep 17 00:00:00 2001 From: Johan Stenberg Date: Mon, 21 Sep 2020 14:22:48 -0700 Subject: [PATCH 03/12] Update per review feedback --- docs/python/clouddiscover.md | 35 +++++++++++++++-------------------- 1 file changed, 15 insertions(+), 20 deletions(-) diff --git a/docs/python/clouddiscover.md b/docs/python/clouddiscover.md index 5aecd7067eb..36dfef39b4d 100644 --- a/docs/python/clouddiscover.md +++ b/docs/python/clouddiscover.md @@ -4,16 +4,16 @@ There are multiple Azure Cloud instances. This includes well-known cloud instanc The clouds differ in which endpoints to use when connecting to it. Finding exactly which endpoints to use is challenging for users. This applies both for public clouds and private instances. For private cloud instances, Microsoft does not know the specific endpoints used by the cloud instance. Additionally cloud instances may or may not be reachable from public internet. -In order to discover which endpoints (DNS names and suffixes) are to be used when connecting to a specific cloud, a disovery endpoint has been introduced. This reduces the amount of information a developer needs in order to connect to a given cloud to knowing a single endpoint. +In order to discover which endpoints (DNS names and suffixes) are to be used when connecting to a specific cloud, a discovery endpoint has been introduced. This reduces the amount of information a developer needs in order to connect to a given cloud to knowing a single endpoint. ## Goals * Simplify configuring for a client connecting to public clouds other than Public Azure. -* Allow use of configuration data retreived from cloud configuration discovery endpoint for private cloud instances. +* Allow use of configuration data retrieved from cloud configuration discovery endpoint for private cloud instances. ## Non-goals -* Automatically discover which cloud an application runs in. +* Automatically discover which cloud an application runs in and default libraries to use that by default. * Change in requirements for client libraries to always accept a simple name (e.g. storage account name) in addition to full endpoint information when constructing clients. ## Core capabilities @@ -86,6 +86,8 @@ None of the properties in a service configuration entry are required as they dif * Service specific client libraries that have baked-in cloud-specific defaults MUST accept a cloud configuration instance in the constructor (or equivalent) for the service client instance. Client libraries MAY also accept a a simple cloud name. +> Examples of cloud specific defaults include hostnames (for services sharing a common DNS name, such as ARM) and host name suffixes (for client libraries that concatenate a "simple" service name with a suffix to build the full DNS name to connect to). + ### Ambient default configuration * A language MAY provide the capability to set the default cloud to use. @@ -97,10 +99,10 @@ None of the properties in a service configuration entry are required as they dif The order of precedence is (in order of decreasing specificity): -* Explicitly provided endpoint parameter passed in value in the method call -* Explicitly provided configuration value from cloud_configuration parameter passed in the method call -* Ambient default cloud (if supported) +* Explicitly provided configuration value from cloud_configuration parameter provided when creating the client. +* Ambient default cloud (if supported) explicitly set by the application. * Endpoint specific environment variables (e.g. `AZURE_AUTHORITY_HOST`) +* Cloud specified in the `AZURE_CLOUD` environment variable. * Final fallback, corresponding Public Azure's settings (built in to the client library) ### Missing configuration properties @@ -117,6 +119,8 @@ Given that more configuration settings can be added over time, you may run into ### Python ```python +from azure.core import CloudConfiguration + # In order to default to switch all default values to match the expected configuration for # the Azure China Cloud. azure.core.settings.set_default_cloud('AzureChinaCloud') @@ -133,23 +137,14 @@ public_azure_creds = azure.identity.DefaultAzureCredential(authority='https://lo # I can also specify a full cloud configuration object: # In order to default to switch all default values to match the expected configuration for # the Azure China Cloud. -config = CloudConfiguration.from_json(requests.get('https://azurestackinstance1.contoso.com/discover'.json())) +config: typing.Mapping[str, CloudConfiguration] = CloudConfiguration.load(requests.get('https://azurestackinstance1.contoso.com/discover'.json())) azure.core.settings.set_default_cloud(config['AzureStack']) -# ...or pass in a custom cloud configuration into a method -creds = azure.identity.DefaultAzureCredential(cloud_configuration=config['PublicAzure']) -``` - -### C# +# ...or pass in a custom cloud configuration instance into a method +creds = azure.identity.DefaultAzureCredential(cloud_configuration=config['AzureStackInstance2']) -```C# -// In order to default to switch all default values to match the expected configuration for -// the Azure China Cloud. -creds = Azure.Core.Identity.DefaultAzureCredentials( - Azure.Core.Identity.DefaultAzureCredentialOptions() { - CloudConfiguration = Azure.Core.CloudConfigurations.AzureChinaCloud - } -); +# ...or by name +creds = azure.identity.DefaultAzureCredential(cloud_configuration='PublicAzure') ``` ## Future evolution/addition of per-service endpoint configuration From 1815cb1fb73fcf3a10b3334dd97733b6b8d02e94 Mon Sep 17 00:00:00 2001 From: "Johan Stenberg (MSFT)" Date: Mon, 21 Sep 2020 14:37:39 -0700 Subject: [PATCH 04/12] Update according to review feedback --- docs/python/clouddiscover.md | 35 +++++++++++++++-------------------- 1 file changed, 15 insertions(+), 20 deletions(-) diff --git a/docs/python/clouddiscover.md b/docs/python/clouddiscover.md index 5aecd7067eb..36dfef39b4d 100644 --- a/docs/python/clouddiscover.md +++ b/docs/python/clouddiscover.md @@ -4,16 +4,16 @@ There are multiple Azure Cloud instances. This includes well-known cloud instanc The clouds differ in which endpoints to use when connecting to it. Finding exactly which endpoints to use is challenging for users. This applies both for public clouds and private instances. For private cloud instances, Microsoft does not know the specific endpoints used by the cloud instance. Additionally cloud instances may or may not be reachable from public internet. -In order to discover which endpoints (DNS names and suffixes) are to be used when connecting to a specific cloud, a disovery endpoint has been introduced. This reduces the amount of information a developer needs in order to connect to a given cloud to knowing a single endpoint. +In order to discover which endpoints (DNS names and suffixes) are to be used when connecting to a specific cloud, a discovery endpoint has been introduced. This reduces the amount of information a developer needs in order to connect to a given cloud to knowing a single endpoint. ## Goals * Simplify configuring for a client connecting to public clouds other than Public Azure. -* Allow use of configuration data retreived from cloud configuration discovery endpoint for private cloud instances. +* Allow use of configuration data retrieved from cloud configuration discovery endpoint for private cloud instances. ## Non-goals -* Automatically discover which cloud an application runs in. +* Automatically discover which cloud an application runs in and default libraries to use that by default. * Change in requirements for client libraries to always accept a simple name (e.g. storage account name) in addition to full endpoint information when constructing clients. ## Core capabilities @@ -86,6 +86,8 @@ None of the properties in a service configuration entry are required as they dif * Service specific client libraries that have baked-in cloud-specific defaults MUST accept a cloud configuration instance in the constructor (or equivalent) for the service client instance. Client libraries MAY also accept a a simple cloud name. +> Examples of cloud specific defaults include hostnames (for services sharing a common DNS name, such as ARM) and host name suffixes (for client libraries that concatenate a "simple" service name with a suffix to build the full DNS name to connect to). + ### Ambient default configuration * A language MAY provide the capability to set the default cloud to use. @@ -97,10 +99,10 @@ None of the properties in a service configuration entry are required as they dif The order of precedence is (in order of decreasing specificity): -* Explicitly provided endpoint parameter passed in value in the method call -* Explicitly provided configuration value from cloud_configuration parameter passed in the method call -* Ambient default cloud (if supported) +* Explicitly provided configuration value from cloud_configuration parameter provided when creating the client. +* Ambient default cloud (if supported) explicitly set by the application. * Endpoint specific environment variables (e.g. `AZURE_AUTHORITY_HOST`) +* Cloud specified in the `AZURE_CLOUD` environment variable. * Final fallback, corresponding Public Azure's settings (built in to the client library) ### Missing configuration properties @@ -117,6 +119,8 @@ Given that more configuration settings can be added over time, you may run into ### Python ```python +from azure.core import CloudConfiguration + # In order to default to switch all default values to match the expected configuration for # the Azure China Cloud. azure.core.settings.set_default_cloud('AzureChinaCloud') @@ -133,23 +137,14 @@ public_azure_creds = azure.identity.DefaultAzureCredential(authority='https://lo # I can also specify a full cloud configuration object: # In order to default to switch all default values to match the expected configuration for # the Azure China Cloud. -config = CloudConfiguration.from_json(requests.get('https://azurestackinstance1.contoso.com/discover'.json())) +config: typing.Mapping[str, CloudConfiguration] = CloudConfiguration.load(requests.get('https://azurestackinstance1.contoso.com/discover'.json())) azure.core.settings.set_default_cloud(config['AzureStack']) -# ...or pass in a custom cloud configuration into a method -creds = azure.identity.DefaultAzureCredential(cloud_configuration=config['PublicAzure']) -``` - -### C# +# ...or pass in a custom cloud configuration instance into a method +creds = azure.identity.DefaultAzureCredential(cloud_configuration=config['AzureStackInstance2']) -```C# -// In order to default to switch all default values to match the expected configuration for -// the Azure China Cloud. -creds = Azure.Core.Identity.DefaultAzureCredentials( - Azure.Core.Identity.DefaultAzureCredentialOptions() { - CloudConfiguration = Azure.Core.CloudConfigurations.AzureChinaCloud - } -); +# ...or by name +creds = azure.identity.DefaultAzureCredential(cloud_configuration='PublicAzure') ``` ## Future evolution/addition of per-service endpoint configuration From 1e567720ced48d862c745ff605b89bc59316bc10 Mon Sep 17 00:00:00 2001 From: Johan Stenberg Date: Mon, 7 Dec 2020 10:46:19 -0800 Subject: [PATCH 05/12] Add authentication attribute to per-service config --- docs/python/clouddiscover.md | 50 ++++++++++++++++++++++-------------- 1 file changed, 31 insertions(+), 19 deletions(-) diff --git a/docs/python/clouddiscover.md b/docs/python/clouddiscover.md index 36dfef39b4d..01379ccdc99 100644 --- a/docs/python/clouddiscover.md +++ b/docs/python/clouddiscover.md @@ -36,29 +36,41 @@ In order to discover which endpoints (DNS names and suffixes) are to be used whe A cloud configuration object consists of a map of service-name to service-specific-configuration-entry as per below: -```json +```jsonc { - "": { - "endpoint": "", - "suffix": "", - "audiences": [ - "" - ], - "tenant": "", - "identityProvider": "" - }, - "": { - "endpoint": "", - "suffix": "", - "audiences": [ - "" - ], - "tenant": "", - "identityProvider": "" - } + "": { + "endpoint": "", + "suffix": "", + "authentication": { + "audiences": [ + "" + ], + "tenant": "", + "identityProvider": "" + } + }, + "": { + "endpoint": "", + "suffix": "", + "authentication": { + "audiences": [ + "" + ], + "tenant": "", + "identityProvider": "" + } + } } ``` + +|Property|Description|Example| +|-|-|-| +|endpoint|Absolute URL (including domain name) for the service. Used by multitenant services.|`https://management.microsoftazure.de` +|suffix|Clouds specific domain suffix for the service. Used for single tenant services with unique hostname per instance/account.|`.vault.microsoftazure.de` +|audiences|List of audiences for the given service| +|tenant| + where `` is (currently) one of the following values: |Name|Notes| From 800998db59763cc3fd1a9cecce74be330f01a115 Mon Sep 17 00:00:00 2001 From: Johan Stenberg Date: Mon, 5 Apr 2021 11:06:13 -0700 Subject: [PATCH 06/12] -S --- docs/python/clouddiscover.md | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/docs/python/clouddiscover.md b/docs/python/clouddiscover.md index 01379ccdc99..1779e27b07d 100644 --- a/docs/python/clouddiscover.md +++ b/docs/python/clouddiscover.md @@ -2,7 +2,7 @@ There are multiple Azure Cloud instances. This includes well-known cloud instances such as Public Azure, Azure China Cloud, Azure Government Cloud etc. It also includes private cloud instances that are managed by our customers. An example of this is Azure Stack. -The clouds differ in which endpoints to use when connecting to it. Finding exactly which endpoints to use is challenging for users. This applies both for public clouds and private instances. For private cloud instances, Microsoft does not know the specific endpoints used by the cloud instance. Additionally cloud instances may or may not be reachable from public internet. +The clouds differ in which endpoints to use when connecting to it. Finding exactly which endpoints to use is challenging for users. This applies both for public clouds and private instances. For private cloud instances, Microsoft does not know the specific endpoints used by the cloud instance. Additionally cloud instances may or may not be reachable from public internet. In order to discover which endpoints (DNS names and suffixes) are to be used when connecting to a specific cloud, a discovery endpoint has been introduced. This reduces the amount of information a developer needs in order to connect to a given cloud to knowing a single endpoint. @@ -20,15 +20,17 @@ In order to discover which endpoints (DNS names and suffixes) are to be used whe * Well-known (public) cloud configurations MUST be built-in to the client libraries for each language. Each well-known cloud configuration has a "simple name" in addition to a set of key/value pairs representing endpoints and suffixes for known services. -> The built-in cloud configurations SHOULD be defined in azure core. +> The built-in cloud configurations MUST be defined in azure core. > The simple cloud name MAY be a string or an enum value. -> The cloud configuration instance MAY be represented by a custom type in strongly typed lanugages. +> The cloud configuration instance MAY be represented by a custom type in strongly typed languages. -* In addition to the well-known cloud configurations, it MUST be possible to construct a custom cloud configuration instance for a specific cloud (e.g. Azure Stack). +* In addition to the well-known cloud configurations, it MUST be possible to construct a custom cloud configuration for a specific cloud instance (e.g. Azure Stack). -* It MUST be possible (and documented how) to load a configuration from the returned information in ARM's discovery endpoint. +* The core library MUST provide a mechanism to list well-known cloud instance names. + +* It MUST be possible (and documented how) to load a configuration from the returned information in ARM's discovery endpoint. Unknown configuration data retrieved from the ARM discovery endpoint (e.g. configuration data associated with new services) MUST be ignored. > A canonical use-case for this is to, given the metadata configuration endpoint for a given cloud, download the configuration data to a local file. @@ -63,13 +65,12 @@ A cloud configuration object consists of a map of service-name to service-specif } ``` - |Property|Description|Example| |-|-|-| |endpoint|Absolute URL (including domain name) for the service. Used by multitenant services.|`https://management.microsoftazure.de` |suffix|Clouds specific domain suffix for the service. Used for single tenant services with unique hostname per instance/account.|`.vault.microsoftazure.de` |audiences|List of audiences for the given service| -|tenant| +|tenant|Tenant used to authenticate the given service| where `` is (currently) one of the following values: @@ -88,12 +89,14 @@ where `` is (currently) one of the following values: |`sqlServer`|| |`storage`|| -None of the properties in a service configuration entry are required as they differ between different services. +None of the properties in a service configuration entry are required as they differ between different services. > Note that this format differs from the raw json exposed by the discovery endpoint. The endpoint discovery response is not structured on a per-service basis, which makes it harder to evolve the set of metadata in the response. > A given language can choose to expose the configuration as a dictionary or as a strongly typed class. +* It MUST be possible to persist and load configuration objects. + ## Service specific client library guidance * Service specific client libraries that have baked-in cloud-specific defaults MUST accept a cloud configuration instance in the constructor (or equivalent) for the service client instance. Client libraries MAY also accept a a simple cloud name. @@ -127,9 +130,10 @@ Given that more configuration settings can be added over time, you may run into * Adding a new required configuration property for a given method/API is a breaking change. -## Example usage +## Example usage ### Python + ```python from azure.core import CloudConfiguration @@ -166,4 +170,3 @@ creds = azure.identity.DefaultAzureCredential(cloud_configuration='PublicAzure') ## Issues * Only updated client libraries will pick up the ambient default settings. This problem will fade over time as new client library versions are made aware of ambient settings. - From febd2ea4a1ad77b358ebb9fe9e12ec7925632b03 Mon Sep 17 00:00:00 2001 From: Johan Stenberg Date: Fri, 9 Apr 2021 15:31:53 -0700 Subject: [PATCH 07/12] Update proposal with "global" entry for auth rather than per-service entry --- docs/python/clouddiscover.md | 106 +++++++++++++++++++++++++++++------ 1 file changed, 88 insertions(+), 18 deletions(-) diff --git a/docs/python/clouddiscover.md b/docs/python/clouddiscover.md index 1779e27b07d..0fb88769500 100644 --- a/docs/python/clouddiscover.md +++ b/docs/python/clouddiscover.md @@ -30,6 +30,11 @@ In order to discover which endpoints (DNS names and suffixes) are to be used whe * The core library MUST provide a mechanism to list well-known cloud instance names. +```python +for known_cloud in azure.core.clouds.well_known: + print(known_cloud) +``` + * It MUST be possible (and documented how) to load a configuration from the returned information in ARM's discovery endpoint. Unknown configuration data retrieved from the ARM discovery endpoint (e.g. configuration data associated with new services) MUST be ignored. > A canonical use-case for this is to, given the metadata configuration endpoint for a given cloud, download the configuration data to a local file. @@ -40,15 +45,21 @@ A cloud configuration object consists of a map of service-name to service-specif ```jsonc { + "global": { + "endpoint": "", + "authentication": { + "audiences": [ + "" + ] + } + }, "": { "endpoint": "", "suffix": "", "authentication": { "audiences": [ "" - ], - "tenant": "", - "identityProvider": "" + ] } }, "": { @@ -57,9 +68,7 @@ A cloud configuration object consists of a map of service-name to service-specif "authentication": { "audiences": [ "" - ], - "tenant": "", - "identityProvider": "" + ] } } } @@ -76,20 +85,21 @@ where `` is (currently) one of the following values: |Name|Notes| |-|-| -|`acr`|Azure Container Registry| -|`authentication`|| +|`global`|Global per-cloud configuration (e.g. auth endpoint etc.). Required| |`batch`|Batch service endpoint| -|`dataLakeCatalogAndJob`|| -|`dataLakeFileSystem`|| -|`microsoftGraph`|| +|`containerRegistry`|Azure Container Registry| +|`dataLakeAnalyticsCatalogAndJob`|| +|`dataLakeStorageFileSystem`|| +|`graph`|| |`keyvault`|| |`media`|Media services service endpoint| |`portal`|Portal address| |`resourceManager`|| -|`sqlServer`|| +|`sql`|| +|`sqlManagement`|| |`storage`|| -None of the properties in a service configuration entry are required as they differ between different services. +None of then individual properties in a service configuration entry are required as they differ between different services (e.g. a service can have a well-known endpoint or a suffix, but it never has both) > Note that this format differs from the raw json exposed by the discovery endpoint. The endpoint discovery response is not structured on a per-service basis, which makes it harder to evolve the set of metadata in the response. @@ -135,11 +145,11 @@ Given that more configuration settings can be added over time, you may run into ### Python ```python -from azure.core import CloudConfiguration +from azure.core.clouds import CloudConfiguration # In order to default to switch all default values to match the expected configuration for # the Azure China Cloud. -azure.core.settings.set_default_cloud('AzureChinaCloud') +azure.core.settings.cloud_configuration = 'AzureChinaCloud' creds = azure.identity.DefaultAzureCredential() # We will use the Azure China Cloud's authority (https://login.chinacloudapi.cn) since that is what is configured as the default cloud. @@ -153,11 +163,14 @@ public_azure_creds = azure.identity.DefaultAzureCredential(authority='https://lo # I can also specify a full cloud configuration object: # In order to default to switch all default values to match the expected configuration for # the Azure China Cloud. -config: typing.Mapping[str, CloudConfiguration] = CloudConfiguration.load(requests.get('https://azurestackinstance1.contoso.com/discover'.json())) -azure.core.settings.set_default_cloud(config['AzureStack']) +azure.core.settings.cloud_configuration = azure.core.clouds.well_known['AzureChinaCloud'] # ...or pass in a custom cloud configuration instance into a method -creds = azure.identity.DefaultAzureCredential(cloud_configuration=config['AzureStackInstance2']) +config: typing.Mapping[str, CloudConfiguration] = { + data['name']: CloudConfiguration.from_metadata_dict(data) + for data in requests.get('https://azurestackinstance1.contoso.com/discover?api-version=2019-05-01'.json() +} +creds = azure.identity.DefaultAzureCredential(cloud_configuration=config['AzureStackInstance1']) # ...or by name creds = azure.identity.DefaultAzureCredential(cloud_configuration='PublicAzure') @@ -167,6 +180,63 @@ creds = azure.identity.DefaultAzureCredential(cloud_configuration='PublicAzure') * Over time metadata can be added for a given service/new services may be added. Client libraries that depend on the new metadata can set the minimum version of the azure core dependency to ensure that the new metadata is available for well-known clouds. +* New service entries are added to azure-core on demand (i.e. when a client library needs new configuration values) + ## Issues * Only updated client libraries will pick up the ambient default settings. This problem will fade over time as new client library versions are made aware of ambient settings. + +## Appendix + +### Cloud configuration schema + +```json-schema +{ + "$schema": "http://json-schema.org/draft-04/schema#", + "$id": "https://example.com/cloudconfig.schema.json", + "title": "Azure Cloud Configuration", + "description": "Endpoint and authentication information for azure cloud instances", + "type": "object", + "required": [ + "global" + ], + "properties": { + "global": { + "$ref": "#/$defs/CloudConfiguration" + } + }, + "additionalProperties": { + "$ref": "#/$defs/CloudConfiguration" + }, + "$defs": { + "CloudConfiguration": { + "type": "object", + "properties": { + "endpoint": { + "type": "string" + }, + "suffix": { + "type": "string" + }, + "authentication": { + "type": "object", + "required": [ "audiences" ], + "properties": { + "audiences": { + "type": "array", + "items": { + "type": "string" + } + } + } + } + } + }, + "CloudConfigurations": { + "required": [ + "global" + ] + } + } +} +``` From f998286c695b2518f0c6c3e2c89f421cb5a0dc03 Mon Sep 17 00:00:00 2001 From: "Johan Stenberg (MSFT)" Date: Mon, 12 Apr 2021 09:49:53 -0700 Subject: [PATCH 08/12] Update docs/python/clouddiscover.md Co-authored-by: Jonathan Giles --- docs/python/clouddiscover.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/python/clouddiscover.md b/docs/python/clouddiscover.md index 0fb88769500..f819d728618 100644 --- a/docs/python/clouddiscover.md +++ b/docs/python/clouddiscover.md @@ -109,7 +109,7 @@ None of then individual properties in a service configuration entry are required ## Service specific client library guidance -* Service specific client libraries that have baked-in cloud-specific defaults MUST accept a cloud configuration instance in the constructor (or equivalent) for the service client instance. Client libraries MAY also accept a a simple cloud name. +* Service specific client libraries that have baked-in cloud-specific defaults MUST accept a cloud configuration instance in the constructor (or equivalent) for the service client instance. Client libraries MAY also accept a simple cloud name. > Examples of cloud specific defaults include hostnames (for services sharing a common DNS name, such as ARM) and host name suffixes (for client libraries that concatenate a "simple" service name with a suffix to build the full DNS name to connect to). From 43ab51b2dff1b2aa7a3efd903650a89f23081a92 Mon Sep 17 00:00:00 2001 From: "Johan Stenberg (MSFT)" Date: Mon, 12 Apr 2021 09:50:05 -0700 Subject: [PATCH 09/12] Update docs/python/clouddiscover.md Co-authored-by: Jonathan Giles --- docs/python/clouddiscover.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/python/clouddiscover.md b/docs/python/clouddiscover.md index f819d728618..59b874b117d 100644 --- a/docs/python/clouddiscover.md +++ b/docs/python/clouddiscover.md @@ -99,7 +99,7 @@ where `` is (currently) one of the following values: |`sqlManagement`|| |`storage`|| -None of then individual properties in a service configuration entry are required as they differ between different services (e.g. a service can have a well-known endpoint or a suffix, but it never has both) +None of the individual properties in a service configuration entry are required as they differ between different services (e.g. a service can have a well-known endpoint or a suffix, but it never has both) > Note that this format differs from the raw json exposed by the discovery endpoint. The endpoint discovery response is not structured on a per-service basis, which makes it harder to evolve the set of metadata in the response. From 895809d1e400aa275626902faad928e38e5a8181 Mon Sep 17 00:00:00 2001 From: Johan Stenberg Date: Mon, 14 Jun 2021 12:18:00 -0700 Subject: [PATCH 10/12] Update schema of cloud configuration --- docs/python/clouddiscover.md | 114 ++++++++++++++++++++--------------- 1 file changed, 64 insertions(+), 50 deletions(-) diff --git a/docs/python/clouddiscover.md b/docs/python/clouddiscover.md index 59b874b117d..f72747e94c9 100644 --- a/docs/python/clouddiscover.md +++ b/docs/python/clouddiscover.md @@ -41,34 +41,34 @@ for known_cloud in azure.core.clouds.well_known: ### Cloud configuration properties -A cloud configuration object consists of a map of service-name to service-specific-configuration-entry as per below: +A cloud configuration object consists of a common `authentication` sections plus a map of service-name to service-specific-configuration-entry as per below: ```jsonc { - "global": { - "endpoint": "", - "authentication": { - "audiences": [ - "" - ] - } - }, - "": { - "endpoint": "", - "suffix": "", - "authentication": { - "audiences": [ - "" - ] - } + "authentication": { + "loginEndpoint": "", + "audiences": [ + "" + ] }, - "": { - "endpoint": "", - "suffix": "", - "authentication": { - "audiences": [ - "" - ] + "services": { + "": { + "endpoint": "", + "suffix": "", + "authentication": { + "scopes": [ + "" + ] + } + }, + "": { + "endpoint": "", + "suffix": "", + "authentication": { + "scopes": [ + "" + ] + } } } } @@ -85,11 +85,11 @@ where `` is (currently) one of the following values: |Name|Notes| |-|-| -|`global`|Global per-cloud configuration (e.g. auth endpoint etc.). Required| |`batch`|Batch service endpoint| |`containerRegistry`|Azure Container Registry| |`dataLakeAnalyticsCatalogAndJob`|| |`dataLakeStorageFileSystem`|| +|`gallery`|| |`graph`|| |`keyvault`|| |`media`|Media services service endpoint| @@ -126,8 +126,8 @@ The order of precedence is (in order of decreasing specificity): * Explicitly provided configuration value from cloud_configuration parameter provided when creating the client. * Ambient default cloud (if supported) explicitly set by the application. -* Endpoint specific environment variables (e.g. `AZURE_AUTHORITY_HOST`) -* Cloud specified in the `AZURE_CLOUD` environment variable. +* Endpoint specific environment variables (i.e. `AZURE_AUTHORITY_HOST`) +* Cloud specified in the `AZURE_CLOUD` environment variable. * Final fallback, corresponding Public Azure's settings (built in to the client library) ### Missing configuration properties @@ -168,7 +168,7 @@ azure.core.settings.cloud_configuration = azure.core.clouds.well_known['AzureChi # ...or pass in a custom cloud configuration instance into a method config: typing.Mapping[str, CloudConfiguration] = { data['name']: CloudConfiguration.from_metadata_dict(data) - for data in requests.get('https://azurestackinstance1.contoso.com/discover?api-version=2019-05-01'.json() + for data in requests.get('https://some.cloud.instance/metadata/endpoints?api-version=2019-05-01 '.json() } creds = azure.identity.DefaultAzureCredential(cloud_configuration=config['AzureStackInstance1']) @@ -198,17 +198,42 @@ creds = azure.identity.DefaultAzureCredential(cloud_configuration='PublicAzure') "description": "Endpoint and authentication information for azure cloud instances", "type": "object", "required": [ - "global" + "authentication", + "services" ], "properties": { - "global": { - "$ref": "#/$defs/CloudConfiguration" + "authentication": { + "$ref": "#/$defs/Authentication" + }, + "services": { + "additionalProperties": { + "$ref": "#/$defs/CloudConfiguration" + } } }, - "additionalProperties": { - "$ref": "#/$defs/CloudConfiguration" - }, "$defs": { + "Authentication": { + "type": "object", + "required": [ "loginEndpoint", "audiences", "tenant" ], + "properties": { + "loginEndpoint: { + "type": "string" + }, + "audiences": { + "type": "array", + "items": { + "type": "string" + } + }, + "tenant: { + "type": "string" + }, + "identityProvider: { + "default": "AAD", + "type": "string" + } + } + }, "CloudConfiguration": { "type": "object", "properties": { @@ -218,24 +243,13 @@ creds = azure.identity.DefaultAzureCredential(cloud_configuration='PublicAzure') "suffix": { "type": "string" }, - "authentication": { - "type": "object", - "required": [ "audiences" ], - "properties": { - "audiences": { - "type": "array", - "items": { - "type": "string" - } - } - } + "scopes" : { + "type": "array", + "items": { + "type": "string" + } } } - }, - "CloudConfigurations": { - "required": [ - "global" - ] } } } From ba2da3c8d6273c77d24b947ea6352970c9bee47f Mon Sep 17 00:00:00 2001 From: "Johan Stenberg (MSFT)" Date: Wed, 16 Jun 2021 16:41:29 -0700 Subject: [PATCH 11/12] Update clouddiscover.md Change scope to audience --- docs/python/clouddiscover.md | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/docs/python/clouddiscover.md b/docs/python/clouddiscover.md index f72747e94c9..15e207d46cb 100644 --- a/docs/python/clouddiscover.md +++ b/docs/python/clouddiscover.md @@ -46,18 +46,15 @@ A cloud configuration object consists of a common `authentication` sections plus ```jsonc { "authentication": { - "loginEndpoint": "", - "audiences": [ - "" - ] + "loginEndpoint": "" }, "services": { "": { "endpoint": "", "suffix": "", "authentication": { - "scopes": [ - "" + "audiences": [ + "" ] } }, @@ -65,8 +62,8 @@ A cloud configuration object consists of a common `authentication` sections plus "endpoint": "", "suffix": "", "authentication": { - "scopes": [ - "" + "audiences": [ + "" ] } } @@ -243,7 +240,7 @@ creds = azure.identity.DefaultAzureCredential(cloud_configuration='PublicAzure') "suffix": { "type": "string" }, - "scopes" : { + "audiences" : { "type": "array", "items": { "type": "string" From 427ebc8eb6a2e11d49756fbad4aa348d9f282c1d Mon Sep 17 00:00:00 2001 From: "Johan Stenberg (MSFT)" Date: Tue, 6 Jul 2021 10:19:03 -0700 Subject: [PATCH 12/12] Apply suggestions from code review Co-authored-by: Jon Gallant <2163001+jongio@users.noreply.github.com> --- docs/python/clouddiscover.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/python/clouddiscover.md b/docs/python/clouddiscover.md index 15e207d46cb..8207287bd52 100644 --- a/docs/python/clouddiscover.md +++ b/docs/python/clouddiscover.md @@ -4,16 +4,16 @@ There are multiple Azure Cloud instances. This includes well-known cloud instanc The clouds differ in which endpoints to use when connecting to it. Finding exactly which endpoints to use is challenging for users. This applies both for public clouds and private instances. For private cloud instances, Microsoft does not know the specific endpoints used by the cloud instance. Additionally cloud instances may or may not be reachable from public internet. -In order to discover which endpoints (DNS names and suffixes) are to be used when connecting to a specific cloud, a discovery endpoint has been introduced. This reduces the amount of information a developer needs in order to connect to a given cloud to knowing a single endpoint. +To discover which endpoints (DNS names and suffixes) are to be used when connecting to a specific cloud, a discovery endpoint will be introduced. This reduces the amount of information a developer needs to connect to a given cloud to knowing a single endpoint. ## Goals -* Simplify configuring for a client connecting to public clouds other than Public Azure. +* Simplify configuring for a client connecting to clouds other than Public Azure. * Allow use of configuration data retrieved from cloud configuration discovery endpoint for private cloud instances. ## Non-goals -* Automatically discover which cloud an application runs in and default libraries to use that by default. +* Automatically discover which cloud an application runs in and set that as a default for libraries to use. * Change in requirements for client libraries to always accept a simple name (e.g. storage account name) in addition to full endpoint information when constructing clients. ## Core capabilities