[Core] Add workaround for cross-tenant authentication with Track 2 SDKs #16797

jiasli · 2021-02-04T07:58:10Z

Fix #16691: az vnet peering create no longer works cross-tenant

⚠ This is only a temporary workaround for Azure/azure-sdk-for-python#8313.

Changes

Manually add external tokens to x-ms-authorization-auxiliary header.

Regarding auto-refresh

As far as I know and also tested for asynchronous Azure operations (long-running operations)

az network vnet peering create 
    --name myVnetAToMyVnetB
    --resource-group myResourceGroupA
    --vnet-name myVnetA 
    --remote-vnet /subscriptions/0b1f6471-1bf0-4dda-aec3-cb9272f09590/resourceGroups/myResourceGroupB/providers/Microsoft.Network/VirtualNetworks/myVnetB 
    --allow-vnet-access 
    --debug

x-ms-authorization-auxiliary is only necessary in the first request:

azure.core.pipeline.policies._universal: Request URL: 'https://management.azure.com/subscriptions/414af076-009b-4282-9a0a-acf75bcb037e/resourceGroups/myResourceGroupA/providers/Microsoft.Network/virtualNetworks/myVnetA/virtualNetworkPeerings/myVnetAToMyVnetB?api-version=2020-07-01'
azure.core.pipeline.policies._universal: Request method: 'PUT'
azure.core.pipeline.policies._universal: Request headers:
azure.core.pipeline.policies._universal:     'x-ms-authorization-auxiliary': 'Bearer **********'
azure.core.pipeline.policies._universal:     'Authorization': '*****'
azure.core.pipeline.policies._universal: Request body:
azure.core.pipeline.policies._universal: ... "remoteVirtualNetwork": {"id": "/subscriptions/0b1f6471-1bf0-4dda-aec3-cb9272f09590/resourceGroups/myResourceGroupB/providers/Microsoft.Network/VirtualNetworks/myVnetB"} ...

azure.core.pipeline.policies._universal: Response status: 200
azure.core.pipeline.policies._universal: Response headers:
azure.core.pipeline.policies._universal:     'Azure-AsyncOperation': 'https://management.azure.com/subscriptions/414af076-009b-4282-9a0a-acf75bcb037e/providers/Microsoft.Network/locations/eastus/operations/78792a4b-d45f-4037-8149-18baa4fbff5e?api-version=2020-07-01'

_{The response is a 200 with Azure-AsyncOperation, which contradicts Track asynchronous Azure operations. (Azure/azure-rest-api-specs#12828)}

The following pooling request won't need x-ms-authorization-auxiliary and can also succeed:

GET https://management.azure.com/subscriptions/<subId>/providers/Microsoft.Network/locations/eastus/operations/7de4e22f-7876-4596-9857-bcee50d703da?api-version=2020-07-01
Authorization: Bearer *****

200
{"status": "Succeeded"}

Therefore, we can believe tokens in x-ms-authorization-auxiliary is only used once, so auto-refresh within a long-running operation won't be necessary.

Alternative solutions

Develop a custom policy ExternalBearerTokenCredentialPolicy and add it to the client. However, adding a custom policy to a client is current impracticable (Azure/azure-sdk-for-python#16519).

yonzhan · 2021-02-04T09:57:45Z

Core

evelyn-ys · 2021-02-07T03:28:19Z

src/azure-cli-core/azure/cli/core/commands/client_factory.py

+            if external_tenant_tokens:
+                # Hard-code scheme to 'Bearer' as _BearerTokenCredentialPolicyBase._update_headers does.
+                client_kwargs['headers']['x-ms-authorization-auxiliary'] = \
+                    ', '.join("Bearer {}".format(t[1]) for t in external_tenant_tokens)


Can both , and ; be used as the delimiter for each token?

I simply don't know. We need to do some test.

The exiting code

azure-cli/src/azure-cli-core/azure/cli/core/adal_authentication.py

Line 142 in d07ee66

aux_tokens = ';'.join(['{} {}'.format(scheme2, tokens2) for scheme2, tokens2, _ in external_tenant_tokens])

contradicts the document Authenticate requests across tenants

Bearer <auxiliary-token1>, EncryptedBearer <auxiliary-token2>, Bearer <auxiliary-token3>

That's why this PR is still marked as a draft. 😉

According to my experiment of ARM deployment, the correct delimiter should be , .
When the delimiter is ;, the service returns the following error:

{"error":{"code":"InvalidAuxiliaryTokens","message":"Authentication failed for auxiliary token: 'The auxiliary tokens should have proper format \"'scheme' 'value'\". The '1' auxiliary token(s) were not found in proper format. Invalid tokens are in the response header.'"}}

True! Also verified with:

> az rest --url "https://management.azure.com/subscriptions/0b1f6471-1bf0-4dda-aec3-cb9272f09590/resourcegroups?api-version=2020-10-01" --headers "x-ms-authorization-auxiliary=Bearer token1; Bearer token2" --verbose Request URL: 'https://management.azure.com/subscriptions/0b1f6471-1bf0-4dda-aec3-cb9272f09590/resourcegroups?api-version=2020-10-01' Request method: 'GET' Request headers: 'Authorization': 'Bearer eyJ0eXAiOiJKV...' 'x-ms-authorization-auxiliary': 'Bearer token1; Bearer token2' ... Response status: 401 Response headers: 'WWW-Authenticate': 'Bearer authorization_uri="https://login.windows.net/54826b22-38d6-4fb2-bad9-b7b93a3e9c5a", error="invalid_token", error_description="Invalid auxiliary tokens \'Bearer token1; Bearer token2\'."' ... Response content: {"error":{"code":"InvalidAuxiliaryTokens","message":"Authentication failed for auxiliary token: 'The auxiliary tokens should have proper format \"'scheme' 'value'\". The '1' auxiliary token(s) were not found in proper format. Invalid tokens are in the response header.'"}} Unauthorized({"error":{"code":"InvalidAuxiliaryTokens","message":"Authentication failed for auxiliary token: 'The auxiliary tokens should have proper format \"'scheme' 'value'\". The '1' auxiliary token(s) were not found in proper format. Invalid tokens are in the response header.'"}})

This means the doc is right and the current CLI code is wrong.

jiasli · 2021-03-01T05:15:07Z

@jsntcy @msyyc, please kindly test for Network module and add some LiveScenarioTests.

I have already written some testing script: #16691 (comment)

You may refer to VMCrossTenantUpdateScenarioTest to write similar tests:

azure-cli/src/azure-cli/azure/cli/command_modules/vm/tests/latest/test_vm_commands.py

Lines 4877 to 4913 in fa7939b

    
           class VMCrossTenantUpdateScenarioTest(LiveScenarioTest): 
        
               @ResourceGroupPreparer(name_prefix='cli_test_vm_cross_tenant_', location='westus2') 
        
               def test_vm_cross_tenant_update(self, resource_group): 
        
                   self.kwargs.update({ 
        
                       'location': 'westus2', 
        
                       'another_rg': self.create_random_name('cli_test_vm_cross_tenant_', 40), 
        
                       'another_vm': self.create_random_name('cli_test_vm_cross_tenant_', 40), 
        
                       'image_name': self.create_random_name('cli_test_vm_cross_tenant_', 40), 
        
                       'aux_sub': '1c638cf4-608f-4ee6-b680-c329e824c3a8', 
        
                       'aux_tenant': '72f988bf-86f1-41af-91ab-2d7cd011db47', 
        
                       'vm': self.create_random_name('cli_test_vm_cross_tenant_', 40) 
        
                   }) 
        
                   # Prepare sig in another tenant 
        
                   self.cmd('group create -g {another_rg} --location {location} --subscription {aux_sub}', 
        
                            checks=self.check('name', self.kwargs['another_rg'])) 
        
                   self.cmd( 
        
                       'vm create -g {another_rg} -n {vm} --image ubuntults --admin-username clitest1 --generate-ssh-key --subscription {aux_sub}') 
        
                   self.cmd( 
        
                       'vm run-command invoke -g {another_rg} -n {vm} --command-id RunShellScript --scripts "echo \'sudo waagent -deprovision+user --force\' | at -M now + 1 minutes" --subscription {aux_sub}') 
        
                   time.sleep(70) 
        
                   self.cmd('vm deallocate -g {another_rg} -n {vm} --subscription {aux_sub}') 
        
                   self.cmd('vm generalize -g {another_rg} -n {vm} --subscription {aux_sub}') 
        
                   res = self.cmd( 
        
                       'image create -g {another_rg} -n {image_name} --source {vm} --subscription {aux_sub}').get_output_in_json() 
        
                   self.kwargs.update({ 
        
                       'image_id': res['id'] 
        
                   }) 
        
                   self.cmd( 
        
                       'vm create -g {rg} -n {vm} --image {image_id} --admin-username clitest1 --generate-ssh-key --nsg-rule NONE') 
        
                   self.cmd('vm update -g {rg} -n {vm} --set tags.tagName=tagValue', checks=[ 
        
                       self.check('tags.tagName', 'tagValue') 
        
                   ]) 
        
                   self.cmd('group delete -n {another_rg} -y --subscription {aux_sub}')

jiasli · 2021-03-19T04:44:39Z

src/azure-cli/azure/cli/command_modules/vm/custom.py

-        resource = cmd.cli_ctx.cloud.endpoints.active_directory_resource_id
-        cred, _, _ = profile.get_login_credentials(resource=resource,
-                                                   aux_subscriptions=aux_subscriptions)
-        _, _, _, external_tokens = cred.get_all_tokens('https://management.azure.com/.default')


By moving this to core, it now works for sovereign cloud (solves #15750 (comment)).

qwordy · 2021-04-06T06:55:24Z

src/azure-cli-core/azure/cli/core/_profile.py

@@ -573,6 +574,10 @@ def get_login_credentials(self, resource=None, subscription_id=None, aux_subscri
                if sub[_TENANT_ID] != account[_TENANT_ID]:
                    external_tenants_info.append(sub[_TENANT_ID])

+        if external_tenants_info and (identity_type or in_cloud_console()):
+            raise CLIError("Cross-tenant authentication is not supported by managed identity and Cloud Shell. "


CLIError is deprecated.

This error is difficult to classify.

qwordy · 2021-04-06T06:56:22Z

src/azure-cli-core/azure/cli/core/adal_authentication.py

@@ -35,7 +35,7 @@ def _get_token(self, sdk_resource=None):
        """
        external_tenant_tokens = None
        try:
-            scheme, token, full_token = self._token_retriever(sdk_resource)
+            scheme, token, token_entry = self._token_retriever(sdk_resource)


Is it just a rename operation? Does the object structure change?

qwordy · 2021-04-06T07:00:38Z

src/azure-cli-core/azure/cli/core/commands/client_factory.py

+        # As a temporary workaround, manually add external tokens to 'x-ms-authorization-auxiliary' header.
+        #   https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/authenticate-multi-tenant
+        if getattr(cred, "_external_tenant_token_retriever", None):
+            _, _, _, external_tenant_tokens = cred.get_all_tokens(*resource_to_scopes(resource))


What is *

See More on Defining Functions and Unpacking Argument Lists.

qwordy · 2021-04-06T07:01:13Z

src/azure-cli-core/azure/cli/core/commands/client_factory.py

+            _, _, _, external_tenant_tokens = cred.get_all_tokens(*resource_to_scopes(resource))
+            # Hard-code scheme to 'Bearer' as _BearerTokenCredentialPolicyBase._update_headers does.
+            client_kwargs['headers']['x-ms-authorization-auxiliary'] = \
+                ', '.join("Bearer {}".format(t[1]) for t in external_tenant_tokens)


Is there a pipeline support?

I will make a new policy and add it to the pipeline in the future, utilizing Azure/azure-sdk-for-python#17340.

src/azure-cli-core/azure/cli/core/commands/client_factory.py

qwordy · 2021-04-06T07:06:14Z

src/azure-cli/azure/cli/command_modules/vm/custom.py

@@ -3363,8 +3345,7 @@ def create_image_version(cmd, resource_group_name, gallery_name, gallery_image_n
        gallery_name=gallery_name,
        gallery_image_name=gallery_image_name,
        gallery_image_version_name=gallery_image_version,
-        gallery_image_version=image_version,
-        headers={'x-ms-authorization-auxiliary': external_bearer_token}


You deleted lots of code in vm/custom.py and didn't add anything. How are external tokens handled now? Is another PR required to fix it?

You deleted lots of code in vm/custom.py and didn't add anything.

The logic is not deleted, but moved to _get_mgmt_service_client.

How are external tokens handled now?

External tokens are handled automatically when aux_subscriptions is provided to _get_mgmt_service_client.

client = _compute_client_factory(cmd.cli_ctx, aux_subscriptions=aux_subscriptions)

Is another PR required to fix it?

It already works. No other PR needed.

qwordy · 2021-04-06T07:38:46Z

Have you run live test for sig commands?

Co-authored-by: Feiyue Yu <iamyfy@163.com>

jiasli · 2021-04-06T08:44:18Z

Have you run live test for sig commands?

No. You may take some time to run it.

qwordy · 2021-04-07T01:56:04Z

Testing results of VM. Fail. Subscription is not found. Investigating.
https://clitestresultstac.blob.core.windows.net/20210406-124636-446567/index.html
https://clitestresultstac.blob.core.windows.net/20210406-124636-446567/vm.report.parallel.html

jiasli · 2021-04-07T03:03:15Z

Testing results of VM. Fail. Subscription is not found. Investigating.

It is because the test relies on Microsoft tenant

azure-cli/src/azure-cli/azure/cli/command_modules/vm/tests/latest/test_vm_commands.py

Line 3846 in ba16edd

'aux_tenant': '72f988bf-86f1-41af-91ab-2d7cd011db47',

but your test account doesn't have permission there.

cross-tenant authentication

88738f2

jiasli requested review from evelyn-ys, fengzhou-msft, houk-ms, jsntcy, Juliehzl, qwordy and yungezz as code owners February 4, 2021 07:58

jiasli marked this pull request as draft February 4, 2021 07:58

yonzhan assigned jiasli Feb 4, 2021

yonzhan requested review from zhoxing-ms, kairu-ms and msyyc February 4, 2021 09:58

yonzhan added this to the S183 milestone Feb 4, 2021

evelyn-ys reviewed Feb 7, 2021

View reviewed changes

Skip get_all_tokens for managed identity

b6cedae

yonzhan modified the milestones: S183 - For Ignite, S184 Feb 26, 2021

Merge branch 'dev' into cross-tenant

b79feff

jiasli marked this pull request as ready for review March 1, 2021 03:05

jiasli mentioned this pull request Mar 11, 2021

x-ms-authorization-auxiliary header should be redacted Azure/azure-sdk-for-python#17271

Closed

zhoxing-ms approved these changes Mar 11, 2021

View reviewed changes

jiasli commented Mar 19, 2021

View reviewed changes

yonzhan modified the milestones: S184, S185 Mar 20, 2021

msyyc approved these changes Mar 30, 2021

View reviewed changes

evelyn-ys approved these changes Apr 1, 2021

View reviewed changes

jsntcy approved these changes Apr 2, 2021

View reviewed changes

qwordy reviewed Apr 6, 2021

View reviewed changes

src/azure-cli-core/azure/cli/core/commands/client_factory.py Outdated Show resolved Hide resolved

qwordy reviewed Apr 6, 2021

View reviewed changes

Update src/azure-cli-core/azure/cli/core/commands/client_factory.py

6071c37

Co-authored-by: Feiyue Yu <iamyfy@163.com>

qwordy mentioned this pull request Apr 7, 2021

az sig image-version create error, managed identity #16628

Closed

qwordy approved these changes Apr 9, 2021

View reviewed changes

jiasli merged commit 039fa21 into Azure:dev Apr 9, 2021

jiasli deleted the cross-tenant branch April 9, 2021 04:25

jiasli mentioned this pull request Apr 21, 2021

{Core} Fix: In Cloud Shell, cross-tenant authentication fails for user or Service Principal account #17805

Merged

jiasli mentioned this pull request Jul 13, 2023

az vnet peering create no longer works cross-tenant #16691

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Core] Add workaround for cross-tenant authentication with Track 2 SDKs #16797

[Core] Add workaround for cross-tenant authentication with Track 2 SDKs #16797

jiasli commented Feb 4, 2021

yonzhan commented Feb 4, 2021

evelyn-ys Feb 7, 2021

jiasli Feb 8, 2021

zhoxing-ms Mar 11, 2021

jiasli Mar 11, 2021

jiasli commented Mar 1, 2021

jiasli Mar 19, 2021

qwordy Apr 6, 2021

jiasli Apr 6, 2021

qwordy Apr 6, 2021

jiasli Apr 6, 2021

qwordy Apr 6, 2021

jiasli Apr 6, 2021 •

edited

Loading

qwordy Apr 6, 2021

jiasli Apr 6, 2021

qwordy Apr 6, 2021

jiasli Apr 6, 2021

qwordy commented Apr 6, 2021

jiasli commented Apr 6, 2021

qwordy commented Apr 7, 2021 •

edited

Loading

jiasli commented Apr 7, 2021

[Core] Add workaround for cross-tenant authentication with Track 2 SDKs #16797

[Core] Add workaround for cross-tenant authentication with Track 2 SDKs #16797

Conversation

jiasli commented Feb 4, 2021

Changes

Regarding auto-refresh

Alternative solutions

yonzhan commented Feb 4, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jiasli commented Mar 1, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jiasli Apr 6, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qwordy commented Apr 6, 2021

jiasli commented Apr 6, 2021

qwordy commented Apr 7, 2021 • edited Loading

jiasli commented Apr 7, 2021

jiasli Apr 6, 2021 •

edited

Loading

qwordy commented Apr 7, 2021 •

edited

Loading