Migrate modules to Microsoft Graph #22174

jiasli · 2022-04-25T07:29:02Z

Modules to migrate

modules	`az ad`	`az role`	`azure-graphrbac` SDK	Owner	Progress
acs	✔️		✔️	Service: @FumingZhang CLI: @zhoxing-ms	✔️ #22361 Fixed incorrect usage of `identifierUri`, # 22649
ams	✔️		✔️		✔️ #22361, #22703
appconfig		✔️		Service: @pratiksanglikar CLI: @zhoxing-ms	✔️ #22361
appservice		✔️	✔️	Service: @panchagnula CLI: @zhoxing-ms	✔️ #22819
aro			✔️		✔️ #22549
cosmosdb		✔️			✔️ #22432
deploymentmanager		✔️			✔️ Only called when `not is_playback`
eventhubs			✔️ via `az keyvault`		✔️ #22361
hdinsight			✔️		✔️ #22503
iot		✔️		Service: @digimaun CLI: @zhoxing-ms	✔️ #22262
keyvault	✔️	✔️	✔️	CLI: @evelyn-ys	✔️ #22188, #22337
lab			✔️		✔️ #29889
network		✔️			✔️ Only appear in `@live_only`
profile	✔️				✔️ Only appear in help/error message
resource	✔️	✔️		CLI: @zhoxing-ms @cxznmhdcxz	✔️ #22302
servicebus			✔️ via `az keyvault`		✔️ #22361
serviceconnector	✔️				✔️ #22361
servicefabric			✔️	Service: @a-santamaria CLI: @zhoxing-ms	✔️ #28105
sql			✔️ via `az keyvault`		✔️ #22432
storage	✔️	✔️			✔️ Only appear in `LiveScenarioTest`
synapse	✔️	✔️	✔️		✔️ #23098
vm		✔️		CLI: @zhoxing-ms @cxznmhdcxz	✔️ #22303

⏳: SDK migration can be delayed

Work items

For modules calling `az ad` or `az role` commands

Related YAMLs should be re-recorded.
Input arguments and output JSONs should be carefully inspected. Corresponding code should be updated accordingly.
⚠️ This must be finished before 2022-05-18.

For modules calling `azure-graphrbac` SDK

Related code should use azure.cli.command_modules.role._msgrpah.GraphClient to invoke Microsoft Graph API. Detailed documentation for GraphClient is at https://github.com/Azure/azure-cli/blob/dev/doc/microsoft_graph_client.md
The output is now a dict, instead of SDK msrest.serialization.Model object. Change your code accordingly
⚠️ It is recommended to complete the migration before the end of this year, 2022, as explained by https://docs.microsoft.com/en-us/graph/migrate-azure-ad-graph-overview

References

Input argument breaking changes: https://docs.microsoft.com/en-us/cli/azure/microsoft-graph-migration
Output object breaking changes: https://docs.microsoft.com/en-us/graph/migrate-azure-ad-graph-property-differences

Additional information

az ad command invocations are searched using regex cmd\(['"]ad|az ad
az role command invocations are searched using regex cmd\(['"]role|az role
azure-graphrbac SDK invocations are searched using azure.graphrbac

The text was updated successfully, but these errors were encountered:

yonzhan · 2022-04-25T08:50:36Z

Microsoft Graph migration

navba-MSFT · 2022-05-02T04:33:34Z

@jiasli @yonzhan Removing the CXP attention label since the SDK repo team is working actively on this.

jiasli · 2022-05-11T13:09:41Z

Test failures in CI:

https://dev.azure.com/azure-sdk/public/_build/results?buildId=1563340&view=logs&j=4d9a7583-3a39-5165-718c-04fb813e465f&t=6c30694c-16d1-5996-0668-2b6e3a9f9dd5

=========================== short test summary info ============================
FAILED src/azure-cli/azure/cli/command_modules/vm/tests/latest/test_image_builder_commands.py::ImageTemplateTest::test_image_build_managed_image
FAILED src/azure-cli/azure/cli/command_modules/vm/tests/latest/test_image_builder_commands.py::ImageTemplateTest::test_defer_only_commands
FAILED src/azure-cli/azure/cli/command_modules/vm/tests/latest/test_image_builder_commands.py::ImageTemplateTest::test_image_builder_basic
FAILED src/azure-cli/azure/cli/command_modules/vm/tests/latest/test_image_builder_commands.py::ImageTemplateTest::test_image_build_shared_image
FAILED src/azure-cli/azure/cli/command_modules/vm/tests/latest/test_image_builder_commands.py::ImageTemplateTest::test_image_builder_cancel
FAILED src/azure-cli/azure/cli/command_modules/vm/tests/latest/test_image_builder_commands.py::ImageTemplateTest::test_image_builder_basic_sig
FAILED src/azure-cli/azure/cli/command_modules/vm/tests/latest/test_image_builder_commands.py::ImageTemplateTest::test_image_template_outputs
FAILED src/azure-cli/azure/cli/command_modules/vm/tests/latest/test_image_builder_commands.py::ImageTemplateTest::test_image_builder_customizers
FAILED src/azure-cli/azure/cli/command_modules/vm/tests/latest/test_vm_commands.py::VMGalleryImage::test_create_image_version_with_region_cvm_encryptio
FAILED src/azure-cli/azure/cli/command_modules/vm/tests/latest/test_vm_commands.py::VMGalleryImage::test_gallery_e2e
FAILED src/azure-cli/azure/cli/command_modules/vm/tests/latest/test_vm_commands.py::DiskEncryptionSetTest::test_disk_encryption_set_disk_update
FAILED src/azure-cli/azure/cli/command_modules/vm/tests/latest/test_vm_commands.py::DiskEncryptionSetTest::test_disk_encryption_set_snapshot
FAILED src/azure-cli/azure/cli/command_modules/vm/tests/latest/test_vm_commands.py::DiskEncryptionSetTest::test_disk_encryption_set
FAILED src/azure-cli/azure/cli/command_modules/vm/tests/latest/test_vm_commands.py::DiskEncryptionSetTest::test_disk_encryption_set_update
FAILED src/azure-cli/azure/cli/command_modules/ams/tests/latest/test_ams_sp_scenarios.py::AmsSpTests::test_ams_sp_create_reset
FAILED src/azure-cli/azure/cli/command_modules/sql/tests/latest/test_sql_commands.py::SqlManagedInstanceTransparentDataEncryptionScenarioTest::test_sql_mi_tdebyok
FAILED src/azure-cli/azure/cli/command_modules/resource/tests/latest/test_resource.py::PolicyScenarioTest::test_resource_policy_identity
FAILED src/azure-cli/azure/cli/command_modules/resource/tests/latest/test_resource.py::PolicyScenarioTest::test_resource_policy_identity_systemassigned
FAILED src/azure-cli/azure/cli/command_modules/resource/tests/latest/test_resource.py::ManagedAppDefinitionScenarioTest::test_managedappdef
FAILED src/azure-cli/azure/cli/command_modules/resource/tests/latest/test_resource.py::ManagedAppDefinitionScenarioTest::test_managedappdef_inline
FAILED src/azure-cli/azure/cli/command_modules/resource/tests/latest/test_resource.py::ManagedAppScenarioTest::test_managedapp
FAILED src/azure-cli/azure/cli/command_modules/acs/tests/latest/test_aks_commands.py::AzureKubernetesServiceScenarioTest::test_aks_create_default_service_without_skip_role_assignment
FAILED src/azure-cli/azure/cli/command_modules/iot/tests/latest/test_iot_commands.py::IoTHubTest::test_hub_file_upload
FAILED src/azure-cli/azure/cli/command_modules/iot/tests/latest/test_iot_commands.py::IoTHubTest::test_identity_hub
FAILED src/azure-cli/azure/cli/command_modules/cosmosdb/tests/latest/test_cosmosdb_cassandrami_scenario.py::ManagedCassandraScenarioTest::test_managed_cassandra_cluster_without_datacenters
FAILED src/azure-cli/azure/cli/command_modules/cosmosdb/tests/latest/test_cosmosdb_cassandrami_scenario.py::ManagedCassandraScenarioTest::test_managed_cassandra_verify_lists
FAILED src/azure-cli/azure/cli/command_modules/cosmosdb/tests/latest/test_cosmosdb_commands.py::CosmosDBTests::test_cosmosdb_key_vault_key_uri
FAILED src/azure-cli/azure/cli/command_modules/cosmosdb/tests/latest/test_cosmosdb_commands.py::CosmosDBTests::test_cosmosdb_managed_service_identity
FAILED src/azure-cli/azure/cli/command_modules/appconfig/tests/latest/test_appconfig_commands.py::AppConfigMgmtScenarioTest::test_azconfig_mgmt
FAILED src/azure-cli/azure/cli/command_modules/eventhubs/tests/latest/test_eventhub_commands_encryption_test.py::EHNamespaceMSITesting::test_eh_namespace_encryption
FAILED src/azure-cli/azure/cli/command_modules/servicebus/tests/latest/test_servicebus_encryption_commands.py::SBNamespaceMSITesting::test_sb_namespace_encryption
========== 31 failed, 2339 passed, 229 skipped in 1214.27s (0:20:14) ===========

I haven't investigated why these modules' tests didn't fail:

deploymentmanager
network
storage

Update: Since Graph-related commands are only called during live runs.

jiasli · 2022-05-12T13:39:51Z

cosmosdb

Can't record cosmosdb tests because there are hard-coded SPs:

azure-cli/src/azure-cli/azure/cli/command_modules/cosmosdb/tests/latest/test_cosmosdb_cassandrami_scenario.py

Line 125 in 30216d0

    
           vnet_resource = self.cmd('az role assignment create --assignee e5007d2c-4b13-4a74-9b6a-605d99f03501 --role 4d97b98b-1d4f-4787-a291-c67834d212e7 --scope {vnet_id}')

jiasli · 2022-05-12T15:30:15Z

acs

acs's test test_aks_create_default_service_without_skip_role_assignment is sending the wrong objectId:

DEBUG    msrest.http_logger:http_logger.py:50 Request URL: 'https://graph.windows.net/54826b22-38d6-4fb2-bad9-b7b93a3e9c5a/getObjectsByObjectIds?api-version=1.6'
DEBUG    msrest.http_logger:http_logger.py:51 Request method: 'POST'
DEBUG    msrest.http_logger:http_logger.py:52 Request headers:
DEBUG    msrest.http_logger:http_logger.py:56     'Accept': 'application/json'
DEBUG    msrest.http_logger:http_logger.py:56     'Content-Type': 'application/json; charset=utf-8'
DEBUG    msrest.http_logger:http_logger.py:56     'accept-language': 'en-US'
DEBUG    msrest.http_logger:http_logger.py:56     'Content-Length': '92'
DEBUG    msrest.http_logger:http_logger.py:56     'User-Agent': 'python/3.10.4 (Windows-10-10.0.19044-SP0) msrest/0.6.21 msrest_azure/0.6.4 azure-graphrbac/0.60.0 Azure-SDK-For-Python AZURECLI/2.36.0'
DEBUG    msrest.http_logger:http_logger.py:57 Request body:
DEBUG    msrest.http_logger:http_logger.py:63 {"objectIds": ["http://clitestjkfxa5dv5pmfbesmq"], "includeDirectoryObjectReferences": true}

DEBUG    msrest.http_logger:http_logger.py:80 Response status: 400
DEBUG    msrest.http_logger:http_logger.py:81 Response headers:
DEBUG    msrest.http_logger:http_logger.py:83     'Cache-Control': 'no-cache'
DEBUG    msrest.http_logger:http_logger.py:83     'Content-Type': 'application/json; odata=minimalmetadata; streaming=true; charset=utf-8'
DEBUG    msrest.http_logger:http_logger.py:83     'x-ms-dirapi-data-contract-version': '1.6'
DEBUG    msrest.http_logger:http_logger.py:83     'Duration': '2457715'
DEBUG    msrest.http_logger:http_logger.py:83     'DataServiceVersion': '3.0;'
DEBUG    msrest.http_logger:http_logger.py:83     'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
DEBUG    msrest.http_logger:http_logger.py:83     'Date': 'Thu, 12 May 2022 15:14:27 GMT'
DEBUG    msrest.http_logger:http_logger.py:83     'x-ms-resource-unit': '3'
DEBUG    msrest.http_logger:http_logger.py:83     'request-id': 'c1425ac7-21da-49cb-8cbf-77b0610e3153'
DEBUG    msrest.http_logger:http_logger.py:83     'ocp-aad-diagnostics-server-name': 'gPnpdqpwyGrDEewN8wjIW0CFeFa4Hzg72WshawB8akU='
DEBUG    msrest.http_logger:http_logger.py:83     'Content-Length': '204'
DEBUG    msrest.http_logger:http_logger.py:83     'client-request-id': '1640947c-d206-11ec-adc8-84a93e63aa78'
DEBUG    msrest.http_logger:http_logger.py:83     'Pragma': 'no-cache'
DEBUG    msrest.http_logger:http_logger.py:83     'Expires': '-1'
DEBUG    msrest.http_logger:http_logger.py:83     'ocp-aad-session-key': 'GHjl1292Gt6HrT9DhFpvHMO0uqFrHpc0bYIm6XJjuGcRF4vbSP9ramPADInpl1cZD4_d8CAKKBCZQ3hTMqkxXEe93mURSaTIOiaO1ajv7WpIPYV3dxFAu1kwsrSFSEtDwUItO7-srxwWUtT_zslEF-lf5NgFGN20OTdqUHrZ9bY.ZYt-BaOpkMFlA_ukGK3kpBGih9TUL2_IpjNeKWQGmLg'
DEBUG    msrest.http_logger:http_logger.py:83     'X-Powered-By': 'ASP.NET'
DEBUG    msrest.http_logger:http_logger.py:83     'Access-Control-Allow-Origin': '*'
DEBUG    msrest.http_logger:http_logger.py:83     'X-AspNet-Version': '4.0.30319'
DEBUG    msrest.http_logger:http_logger.py:86 Response content:
DEBUG    msrest.http_logger:http_logger.py:101 {"odata.error":{"code":"Request_BadRequest","message":{"lang":"en","value":"Invalid GUID:http://clitestjkfxa5dv5pmfbesmq"},"requestId":"c1425ac7-21da-49cb-8cbf-77b0610e3153","date":"2022-05-12T15:14:27"}}

objectIds should be a GUID, instead of identifierUri http://clitestjkfxa5dv5pmfbesmq.

Using non-verified identifierUri has been forbidden long ago (#19892), but the tests are not re-run since then. It didn't fail before because VCRPY doesn't care about the request body which contains http://clitestjkfxa5dv5pmfbesmq. @FumingZhang

jiasli · 2022-05-12T16:38:53Z

sql

I am not able to re-run sql test test_sql_mi_tdebyok:

> azdev test test_sql_mi_tdebyok --live --series
E           knack.util.CLIError: Subscriptions are restricted from provisioning in this region. Please choose a different region. For exceptions to this rule please open a support request with Issue type of 'Service and subscription limits'. See https://docs.microsoft.com/en-us/azure/sql-database/quota-increase-request for more details.

jiasli · 2022-05-13T05:27:14Z

Now we only have these tests that can't be run, as explained above and in the issue description:

https://dev.azure.com/azure-sdk/public/_build/results?buildId=1565422&view=logs&j=4d9a7583-3a39-5165-718c-04fb813e465f&t=6c30694c-16d1-5996-0668-2b6e3a9f9dd5

=========================== short test summary info ============================
FAILED src/azure-cli/azure/cli/command_modules/sql/tests/latest/test_sql_commands.py::SqlManagedInstanceTransparentDataEncryptionScenarioTest::test_sql_mi_tdebyok
FAILED src/azure-cli/azure/cli/command_modules/iot/tests/latest/test_iot_commands.py::IoTHubTest::test_hub_file_upload
FAILED src/azure-cli/azure/cli/command_modules/iot/tests/latest/test_iot_commands.py::IoTHubTest::test_identity_hub
FAILED src/azure-cli/azure/cli/command_modules/cosmosdb/tests/latest/test_cosmosdb_cassandrami_scenario.py::ManagedCassandraScenarioTest::test_managed_cassandra_cluster_without_datacenters
FAILED src/azure-cli/azure/cli/command_modules/cosmosdb/tests/latest/test_cosmosdb_cassandrami_scenario.py::ManagedCassandraScenarioTest::test_managed_cassandra_verify_lists
FAILED src/azure-cli/azure/cli/command_modules/cosmosdb/tests/latest/test_cosmosdb_commands.py::CosmosDBTests::test_cosmosdb_key_vault_key_uri
FAILED src/azure-cli/azure/cli/command_modules/cosmosdb/tests/latest/test_cosmosdb_commands.py::CosmosDBTests::test_cosmosdb_managed_service_identity
=========== 7 failed, 2363 passed, 229 skipped in 1267.99s (0:21:07) ===========

All these tests are disabled by #22361 and need to be re-recorded by service team.

hivyas · 2022-05-18T21:08:35Z

Hello, I tried updating the ams module to use the new graph api which you can see the code for here. But I am running into this error:
azure.cli.core.azclierror.AuthenticationError: AADSTS53000: Device is not in required device state: compliant. Conditional Access policy requires a compliant device, and the device is not compliant. The user must enroll their device with an approved MDM provider like Intune.

The error is happening from this line.

jiasli · 2022-05-19T00:35:04Z

Hi @hivyas, this is because your machine is not compliant. You need to make sure your machine is AD or domain joined, and sync your computer's group policy.

Click Info button, and you will see a Sync button.

hivyas · 2022-05-19T02:11:19Z

Hi @jiasli, thanks for the response! I checked my accounts and resynced but I am still getting the same error.

ghost added AKS az aks/acs/openshift CXP Attention This issue is handled by CXP team. Auto-Assign Auto assign by bot App Services az appservice KeyVault az keyvault labels Apr 25, 2022

ghost assigned evelyn-ys Apr 25, 2022

ghost added this to the Backlog milestone Apr 25, 2022

ghost added the RBAC az role label Apr 25, 2022

ghost assigned jiasli Apr 25, 2022

ghost added Graph az ad CosmosDB az cosmosdb Service Fabric az sf Synapse labels Apr 25, 2022

yonzhan added the feature-request label Apr 25, 2022

jiasli mentioned this issue Apr 26, 2022

Azure Container Apps - 4/28 release Azure/azure-cli-extensions#4726

Merged

3 tasks

evelyn-ys mentioned this issue Apr 26, 2022

{Graph} Rerun keyvault tests related with az ad/role #22188

Merged

3 tasks

zhoxing-ms self-assigned this Apr 26, 2022

evelyn-ys mentioned this issue Apr 27, 2022

{Graph} Drop azure-graphrbac SDK in keyvault module #22203

Merged

3 tasks

navba-MSFT removed the CXP Attention This issue is handled by CXP team. label May 2, 2022

vilit1 mentioned this issue May 4, 2022

{IoT} Test updates for beta branch #22262

Merged

3 tasks

jiasli mentioned this issue May 7, 2022

{Graph} Refactor role module for Microsoft Graph migration #22300

Merged

This was referenced May 7, 2022

{Graph} Re-record related tests of resource module #22302

Merged

{Graph} Re-record related tests of vm module #22303

Merged

jiasli mentioned this issue May 11, 2022

{Graph} Migrate command modules to use Microsoft Graph API #22361

Merged

jiasli added the Microsoft Graph label May 12, 2022

This was referenced May 19, 2022

Migrating to Microsoft Graph for az cli Azure/ARO-RP#2129

Closed

{aro} az aro module migration to Microsoft Graph #22549

Merged

This was referenced May 23, 2022

{Graph} Make msgraph internal and add documentation for GraphClient #22560

Merged

Azure CLI is migrated to Microsoft Graph in 2.37.0 #22580

Closed

hivyas mentioned this issue Jun 2, 2022

{AMS} Migrate AMS module to new Microsoft graph #22703

Merged

3 tasks

This was referenced Jun 3, 2022

az webapp config ssl import fails: AttributeError: 'GraphClient' object has no attribute 'service_principals' #22722

Closed

{AKS} Graph module migration #22649

Merged

kevinzz6 mentioned this issue Jul 4, 2022

{synapse} az synapse module migration to Microsoft Graph #23098

Merged

3 tasks

jiasli mentioned this issue Sep 5, 2024

{Service Fabric} Migrate servicefabric command module to Microsoft Graph #28105

Merged

This was referenced Sep 6, 2024

{AKS} Remove dependency on azure.graphrbac Azure/azure-cli-extensions#7958

Merged

{AKS} Remove dependency on azure.graphrbac #29858

Merged

jiasli mentioned this issue Sep 11, 2024

{Lab} Migrate lab command module to Microsoft Graph #29889

Merged

jiasli closed this as completed Sep 13, 2024

jiasli mentioned this issue Oct 15, 2024

{Packaging} Drop azure-graphrbac SDK #29838

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate modules to Microsoft Graph #22174

Migrate modules to Microsoft Graph #22174

jiasli commented Apr 25, 2022 •

edited

Loading

yonzhan commented Apr 25, 2022

navba-MSFT commented May 2, 2022

jiasli commented May 11, 2022 •

edited

Loading

jiasli commented May 12, 2022 •

edited

Loading

jiasli commented May 12, 2022 •

edited

Loading

jiasli commented May 12, 2022 •

edited

Loading

jiasli commented May 13, 2022 •

edited

Loading

hivyas commented May 18, 2022

jiasli commented May 19, 2022

hivyas commented May 19, 2022

Migrate modules to Microsoft Graph #22174

Migrate modules to Microsoft Graph #22174

Comments

jiasli commented Apr 25, 2022 • edited Loading

Modules to migrate

Work items

For modules calling az ad or az role commands

For modules calling azure-graphrbac SDK

References

Additional information

yonzhan commented Apr 25, 2022

navba-MSFT commented May 2, 2022

jiasli commented May 11, 2022 • edited Loading

jiasli commented May 12, 2022 • edited Loading

cosmosdb

jiasli commented May 12, 2022 • edited Loading

acs

jiasli commented May 12, 2022 • edited Loading

sql

jiasli commented May 13, 2022 • edited Loading

hivyas commented May 18, 2022

jiasli commented May 19, 2022

hivyas commented May 19, 2022

jiasli commented Apr 25, 2022 •

edited

Loading

For modules calling `az ad` or `az role` commands

For modules calling `azure-graphrbac` SDK

jiasli commented May 11, 2022 •

edited

Loading

jiasli commented May 12, 2022 •

edited

Loading

jiasli commented May 12, 2022 •

edited

Loading

jiasli commented May 12, 2022 •

edited

Loading

jiasli commented May 13, 2022 •

edited

Loading