Automate VM MSI testing

[jianghaolu]

We need to automate the way we run tests. The steps to run MSI tests in VM for the identity component are as follows:

prerequisite tools

• Azure CLI

Azure resources

This test requires instances of these Azure resources:

• Azure Key Vault
• Azure Managed Identity
  • with secrets/set and secrets/delete permission for the Key Vault
• Azure Virtual Machine
  • with system assigned identity and user assigned identity

Initial Setup:

PLEASE NOTE: This setup is Java specific since it installs the Java & Maven environments. Other languages will have a different set up.

Set environment variables to simplify copy-pasting

• RESOURCE_GROUP
  • name of an Azure resource group
  • must be unique in the Azure subscription
  • e.g. 'identity-test-rg'
• VM_NAME_SYSTEM_ASSIGNED
  • name of an Azure Virtual machine
  • must be unique in the resource group
  • e.g. 'identity-test-vm-system'
• VM_NAME_USER_ASSIGNED
  • name of an Azure Virtual machine
  • must be unique in the resource group
  • e.g. 'identity-test-vm-user'
• MANAGED_IDENTITY_NAME
  • 3-128 alphanumeric characters
  • must be unique in the resource group
• KEY_VAULT_NAME
  • 3-24 alphanumeric characters
  • must begin with a letter
  • must be globally unique

Create resources

1. In a terminal with Azure CLI, run

az vm create \
    -n $VM_NAME_SYSTEM_ASSIGNED \
    -g $RESOURCE_GROUP \
    --image Canonical:0001-com-ubuntu-server-focal:20_04-lts-gen2:latest \
    --assign-identity \
    --size Standard_DS1_v2 \
    -l westus2
    --generate-ssh-keys

to create a VM with system assigned identity.
2. Create a user identity for the second VM:

az identity create \
    -n $MANAGED_IDENTITY_NAME \
    -g $RESOURCE_GROUP \
    -l westus2

Get the identity resource id

identityResourceId=`az identity show -g $RESOURCE_GROUP -n $MANAGED_IDENTITY_NAME -o json --query "id" | tr -d '"'`

and then create a VM with this identity assigned:

az vm create \
    -n $VM_NAME_USER_ASSIGNED \
    -g $RESOURCE_GROUP \
    --image Canonical:0001-com-ubuntu-server-focal:20_04-lts-gen2:latest \
    --assign-identity $identityResourceId \
    --size Standard_DS1_v2 \
    -l westus2
    --generate-ssh-keys

3. Create a Key Vault:

az keyvault create  -n $KEY_VAULT_NAME -g $RESOURCE_GROUP

Get the principal ID for the VM with system assigned identity:

principalId=`az vm show -n $VM_NAME_SYSTEM_ASSIGNED -g $RESOURCE_GROUP -o json --query identity.principalId | tr -d '"'`

Allow the VM with system assigned identity to access the key vault secrets:

az keyvault set-policy -n $KEY_VAULT_NAME --object-id $principalId --secret-permissions get list

Get the principal ID for the VM with user assigned identity:

principalId=`az vm show -n $VM_NAME_USER_ASSIGNED -g $RESOURCE_GROUP -o json --query "identity.userAssignedIdentities.*.principalId | [0]" | tr -d '"'`

Allow the VM with user assigned identity to access the key vault secrets:

az keyvault set-policy -n $KEY_VAULT_NAME --object-id $principalId --secret-permissions get list

Add a secret called "secret":

az keyvault secret set --vault-name $KEY_VAULT_NAME -n secret --value "Secret value"

Install Java & Maven on VMs

4. Install Java 8 & Maven 3.x:
   Run on both VMs:

echo -e `az vm run-command invoke \
    --ids /subscriptions/{subscription}/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.Compute/virtualMachines/{vm} \
    --command-id RunShellScript \
    --scripts "sudo apt-get update && sudo apt-get install openjdk-8-jdk -y && sudo apt-get install maven -y" \
    -o json \
    --query "value[0].message"`

Run tests

5. Clone azure-sdk-for-java repo
   Run on both VMs:

echo -e `az vm run-command invoke \
    --ids /subscriptions/{subscription}/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.Compute/virtualMachines/{vm} \
    --command-id RunShellScript \
    --scripts "cd /tmp && git clone https://github.com/Azure/azure-sdk-for-java.git --depth 1" \
    -o json \
    --query "value[0].message"`

6. Build the dependencies:

 echo -e `az vm run-command invoke \
    --ids /subscriptions/{subscription}/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.Compute/virtualMachines/{vm} \
    --command-id RunShellScript \
    --scripts "cd /tmp/azure-sdk-for-java && export AZURE_VAULT_URL=https://$KEY_VAULT_NAME.vault.azure.net && mvn install -pl sdk/identity/azure-identity,sdk/core/azure-core,sdk/core/azure-core-http-netty,sdk/keyvault/azure-security-keyvault-keys,sdk/keyvault/azure-security-keyvault-secrets,sdk/keyvault/azure-security-keyvault-certificates,sdk/core/azure-core-test -T 1 -f pom.xml -Dgpg.skip -Dmaven.javadoc.skip -Drevapi.skip -DskipSpringITs -DskipTests -Dspotbugs.skip -Djacoco.skip -am -DfailIfNoTests=false" \                                                                                                                   -o json \
    --query "value[0].message"`

7. Run the MSI test:
   On the VM with system assigned identity:

echo -e `az vm run-command invoke \
    --ids /subscriptions/{subscription}/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.Compute/virtualMachines/{vm} \
    --command-id RunShellScript \
    --scripts "cd /tmp/azure-sdk-for-java && export AZURE_VAULT_URL=https://$KEY_VAULT_NAME.vault.azure.net && mvn test -Dtest=ManagedIdentityCredentialLiveTest#testIMDSEndpointWithSystemAssigned* -pl sdk/e2e -f pom.client.xml -Dgpg.skip -am -DfailIfNoTests=false" \
    -o json \
    --query "value[0].message"`

On the VM with user assigned identity:

echo -e `az vm run-command invoke \
    --ids /subscriptions/{subscription}/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.Compute/virtualMachines/{vm} \
    --command-id RunShellScript \
    --scripts "cd /tmp/azure-sdk-for-java && export AZURE_VAULT_URL=https://$KEY_VAULT_NAME.vault.azure.net && export AZURE_CLIENT_ID=$clientId && mvn test -Dtest=ManagedIdentityCredentialLiveTest#testIMDSEndpointWithUserAssigned* -pl sdk/e2e -f pom.client.xml -Dgpg.skip -am -DfailIfNoTests=false" \
    -o json \
    --query "value[0].message"`

To find the identity's client id

clientId=`az vm show -n $VM_NAME_USER_ASSIGNED -g $RESOURCE_GROUP -o json --query "identity.userAssignedIdentities.*.clientId | [0]" | tr -d '"'`

Boths result should be printed showing

[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.azure.e2e.identity.ManagedIdentityCredentialLiveTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.911 s - in com.azure.e2e.identity.ManagedIdentityCredentialLiveTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

[kurtzeborn]

@jianghaolu, it looks like you've got most of this figured out, what assistance do you need from the engineering system team?

[catalinaperalta]

For Golang VM testing the set up is the same up to step 3. The steps for installing Go on the VM and then running the tests are as follows:
NOTE: I ran these commands in Powershell

Install Golang on each VM

4. echo (az vm run-command invoke --ids /subscriptions/{subscription}/resourceGroups/{resource group}/providers/Microsoft.Compute/virtualMachines/{vm name} --command-id RunShellScript --scripts "sudo add-apt-repository ppa:longsleep/golang-backports && sudo apt-get update -y && sudo apt-get install golang-go -y" --o json --query "value[0].message")

Clone Go SDK repo

5. echo (az vm run-command invoke --ids /subscriptions/{subscription}/resourceGroups/{resource group}/providers/Microsoft.Compute/virtualMachines/{VM name} --command-id RunShellScript --scripts "mkdir tmp && cd /tmp && git clone -b track2_cloudshell https://github.com/Azure/azure-sdk-for-go.git" -o json --query "value[0].message")

Run tests

6. echo (az vm run-command invoke --ids /subscriptions/{subscription}/resourceGroups/{resource group}/providers/Microsoft.Compute/virtualMachines/{VM name} --command-id RunShellScript --scripts "cd /tmp/azure-sdk-for-go/sdk/azidentity && export AZURE_VAULT_URL=https://kvcpname0.vault.azure.net && go test -run TestManagedIdentityCredential_GetTokenInVMLive" -o json --query "value[0].message")

Expected result (it should say PASS in the message)

"Enable succeeded: \n[stdout]\nPASS\nok \tgithub.com/Azure/azure-sdk-for-go/sdk/azidentity\t0.435s\n\n[stderr]\n"

Please Note

If step 6 returns an error saying that GoCache variable or HOME variables aren't set, then assign the path of the user's home directory as follows:
export HOME=/path/to/home/directory

[chlowell]

Python instructions are here.

[XuGuang-Yao]

Hi @chlowell ,

I am following above steps to do python E2E test. My suggestion as below. Please correct me if I miss anything.

1.Since the test needs to delete secret after setting.

client = SecretClient(live_managed_identity_config["vault_url"], credential, logging_enable=True)
secret = client.set_secret("managed-identity-test-secret", "value")
client.begin_delete_secret(secret.name)

So we need to create a non-soft-delete KV in this step.

az keyvault create -g $RESOURCE_GROUP -n $KEY_VAULT_NAME --sku standard --enable-soft-delete false

If not, the test command will fail with below error.

2.We also need add admin name when creating vm.

az vm create -n $VM_NAME_SYSTEM_ASSIGNED -g $RESOURCE_GROUP --image UbuntuLTS --assign-identity --size Standard_DS1_v2 -l westus2 --generate-ssh-keys --admin-username $VM_ADMIN_USER_NAME

az vm create -n $VM_NAME_USER_ASSIGNED -g $RESOURCE_GROUP --image UbuntuLTS --assign-identity $(az identity show -g $RESOURCE_GROUP -n $MANAGED_IDENTITY_NAME -o tsv --query id) --size Standard_DS1_v2 -l westus2 --generate-ssh-keys --admin-username $VM_ADMIN_USER_NAME

[XuGuang-Yao]

Hi @jianghaolu,

I am following above steps to do Java E2E test. My suggestion as below. Please correct me if I miss anything.

1.In step of creating vm command, we need specify the admin name using below command or error will occur.

az vm create -n $VM_NAME_SYSTEM_ASSIGNED -g $RESOURCE_GROUP --image UbuntuLTS --assign-identity --size Standard_DS1_v2 -l westus2 --generate-ssh-keys --admin-username $VM_ADMIN_USER_NAME

az vm create -n $VM_NAME_USER_ASSIGNED -g $RESOURCE_GROUP --image UbuntuLTS --assign-identity $identityResourceId --size Standard_DS1_v2 -l westus2 --generate-ssh-keys --admin-username $VM_ADMIN_USER_NAME

[jianghaolu]

@XuGuang-Yao Which version of Azure CLI are you using? I'm using 2.0.78 on Mac and the username is automatically using my local user account's name. When I run az vm create -h there's a section:

...
Authentication Arguments
    --admin-password               : Password for the VM if authentication type is 'Password'.
    --admin-username               : Username for the VM.  Default: jianghlu.
...

But I agree with you we probably should be more specific on the username. Some environments may not be able to populate a default value.

[XuGuang-Yao]

@jianghaolu ，Thanks for your quick reply. I am using 2.2.0 on windows. There is something wrong with my default account name, my fault.
Since we talk about environment. Which OS should I use to run these tests? May I continue with windows if we don't have specific one?

[XuGuang-Yao]

@jianghaolu, I tried to repeat the test steps today and found the last command doesn't work as expected.
Error as below picture.

In the lastest verison of repo. I can't find the pom.client.xml any more. I noticed this file highly like the removed pom.client.xml. I tried to use it instead of the pom.client.xml, but it doesn't work either.

Could you please help to do a further investigation and correct the command?

[jianghaolu]

Ahh yes - they changed the pom file structure in the repo. Breaking a lot of people apparently. Let me fix the instructions.

[sophiajt]

For JavaScript:

prerequisite tools

• Azure CLI

Azure resources

This test requires instances of these Azure resources:

• Azure Key Vault
• Azure Managed Identity
  • with secrets/set and secrets/delete permission for the Key Vault
• Azure Virtual Machine with system-assigned identity
• Azure Virtual Machine with user-assigned identity
  • don't use the same VM twice

The rest of this section is a walkthrough of deploying these resources.

Set environment variables to simplify copy-pasting

• RESOURCE_GROUP
  • name of an Azure resource group
  • must be unique in the Azure subscription
  • e.g. 'identity-test-rg'
• VM_NAME_SYSTEM_ASSIGNED
  • name of an Azure Virtual machine with a system-assigned identity
  • must be unique in the resource group
  • e.g. 'identity-test-vm-system'
• VM_NAME_USER_ASSIGNED
  • name of an Azure Virtual machine with a user-assigned identity
  • must be unique in the resource group
  • e.g. 'identity-test-vm-user'
• MANAGED_IDENTITY_NAME
  • name of the user-assigned identity
  • 3-128 alphanumeric characters
  • must be unique in the resource group
• KEY_VAULT_NAME
  • 3-24 alphanumeric characters
  • must begin with a letter
  • must be globally unique

Run inside of PowerShell

These instructions assume you're running inside of PowerShell.

resource group

az group create -n $RESOURCE_GROUP --location westus2

Managed identity

Create the identity:

az identity create -n $MANAGED_IDENTITY_NAME -g $RESOURCE_GROUP -l westus2

Virtual machines

With system-assigned identity:

az vm create -n $VM_NAME_SYSTEM_ASSIGNED -g $RESOURCE_GROUP --image UbuntuLTS --assign-identity --size Standard_DS1_v2 -l westus2 --generate-ssh-keys

With user-assigned identity:

az vm create -n $VM_NAME_USER_ASSIGNED -g $RESOURCE_GROUP --image UbuntuLTS --assign-identity $(az identity show -g $RESOURCE_GROUP -n $MANAGED_IDENTITY_NAME -o tsv --query id) --size Standard_DS1_v2 -l westus2 --generate-ssh-keys

Key Vault:

az keyvault create -g $RESOURCE_GROUP -n $KEY_VAULT_NAME --sku standard

Allow the VM with system-assigned identity to access the Key Vault's secrets:

az keyvault set-policy -n $KEY_VAULT_NAME --object-id $(az vm show -n $VM_NAME_SYSTEM_ASSIGNED -g $RESOURCE_GROUP --query identity.principalId -o tsv) --secret-permissions set delete

Do the same for the user-assigned identity:

az keyvault set-policy -n $KEY_VAULT_NAME --object-id $(az identity show -g $RESOURCE_GROUP -n $MANAGED_IDENTITY_NAME --query principalId -o tsv) --secret-permissions set delete

Install dependencies

get user-assigned client id

$VM_ID_USER_ASSIGNED = az vm show -g $RESOURCE_GROUP -n $VM_NAME_USER_ASSIGNED -o tsv --query "identity.userAssignedIdentities.*.{clientId: clientId}"

Build the webapp

cd azure-sdk-for-js\sdk\identity\identity\test\manual-integration\AzureVM

Install the requirements:

npm install

Build the job:

tsc -p .

Build and run the tests

 az vm run-command invoke -g $RESOURCE_GROUP -n $VM_NAME_USER_ASSIGNED --command-id RunShellScript --parameters "KEY_VAULT_NAME=$($KEY_VAULT_NAME) VM_ID_USER_ASSIGNED=$($VM_ID_USER_ASSIGNED)"--scripts '"sudo apt update && (yes | sudo apt install npm) && npm install -g typescript && git clone https://github.com/azure/azure-sdk-for-js --single-branch --branch master --depth 1 && cd azure-sdk-for-js/sdk/identity/identity/test/manual-integration/AzureVM && npm install && tsc -p ."'

az vm run-command invoke -g $RESOURCE_GROUP -n $VM_NAME_SYSTEM_ASSIGNED --command-id RunShellScript --parameters "KEY_VAULT_NAME=$($KEY_VAULT_NAME)" --scripts '"sudo apt update && (yes | sudo apt install npm) && sudo npm install -g typescript && git clone https://github.com/azure/azure-sdk-for-js --single-branch --branch master --depth 1 && cd azure-sdk-for-js/sdk/identity/identity/test/manual-integration/AzureVM && npm install && tsc -p . && node index"'

Note, if you see a message like "Could not get lock /var/lib/apt/lists/lock - open", you may need to re-run the command.

Verify success

az keyvault secret show -n "secret-name-system" --vault-name "$($KEY_VAULT_NAME)"

az keyvault secret show -n "secret-name-user" --vault-name "$($KEY_VAULT_NAME)"

Delete Azure resources

az group delete -n $RESOURCE_GROUP -y --no-wait

[XuGuang-Yao]

Hi @jonathandturner,

I am following above steps to do JS E2E test. My suggestion as below. Please correct me if I miss anything.

Same keytar issue here.

I tried to add (yes | sudo apt-get install libsecret-1-dev) before npm install as the following command, and the test passed.

az vm run-command invoke -g $RESOURCE_GROUP -n $VM_NAME_USER_ASSIGNED --command-id RunShellScript --parameters "KEY_VAULT_NAME=$($KEY_VAULT_NAME) VM_ID_USER_ASSIGNED=$($VM_ID_USER_ASSIGNED)"--scripts "sudo apt update && (yes | sudo apt install npm) && npm install -g typescript && git clone https://github.com/azure/azure-sdk-for-js --single-branch --branch master --depth 1 && cd azure-sdk-for-js/sdk/identity/identity/test/manual-integration/AzureVM && (yes | sudo apt-get install libsecret-1-dev) && npm install && tsc -p . && node index_user"

I also tried to update the command in other tests (such as Cloud Shell and Pod),but they failed with below error.

sudo: The term 'sudo' is not recognized as the name of a cmdlet, function, script file, or operable program.

[sophiajt]

@XuGuang-Yao - can you describe these to help us understand what is happening?

• The version of Windows you're running
• The terminal you're using (WSL, PowerShell, CMD, etc)

You shouldn't need to use commands like sudo if you're running in Windows. Keytar should be able to detect you're in Windows and switch to that mode.

[XuGuang-Yao]

@jonathandturner - I am using PowerShell on windows 10 to test. Since this test needs create a linux VM, so I think the sudo is necessary.

As for the 'sudo' is not recognized error in other test scenarios(such as CloudShell and Pod), I just tried to fix test steps in the same way as this one. (add (yes | sudo apt-get install libsecret-1-dev) before npm install). But it didn't work as expected and encountored the term 'sudo' is not recognized error . Please forgive my poor knowledge.

[sophiajt]

@XuGuang-Yao - the steps above invoke the VM from Windows rather than ssh'ing into the VM. Were there steps above that needed you to log into the VM and run commands directly? I may need to rewrite some of the instructions to be more clear.

[XuGuang-Yao]

@jonathandturner - Let forget the 'sudo' is not recognized error. We should focus on how to install libsecret for all JS E2E tests so that we can reslove the keytar issue. Once this done, the error is no longer a problem.

BTW, I do use invoke to run command in my OS(windows 10) rather than login into VM, just as your instructions above.

[sophiajt]

@XuGuang-Yao - we may need to add a step where you log into the VM and run:

sudo apt install libsecret-1-0

And then do the build steps from the VM. I was able to run the above, sudo npm install -g typescript, npm install, and tsc -p . and have a runnable example.

[XuGuang-Yao]

@jonathandturner -Thanks for your updating. sudo apt install libsecret-1-0 works in this VM test. I tried to run it in other tests(CloudShell and Pod), but it doesn't work. The sudo can't be recognized in Dockerfile or cloudshell.

BTW, I do run Pod test against Docker lunix container environment.

[XuGuang-Yao]

@jianghaolu
Hi, I am following java instrucation to do E2E test against May release. I noticed you haven't update the run test command to pom.xml. Anyway, I tried to change it to pom.xml during the test. But below error occured.

Did I use the wrong pom file? Could you please take a look and correct the test steps?

[XuGuang-Yao]

@jonathandturner - Hope you are doing well. The sudo apt install libsecret-1-0 does solve the problem in VM test. It would be better that we can add sudo apt install libsecret-1-0 to the JS test instruction above. Could you please do that?

[XuGuang-Yao]

@jianghaolu - Hi, I noticed that the instructions are still using the pom.client.xml file. Could you please find time to solve it? Thanks in advance.

[v-jiaodi]

Hi @catalinaperalta, @chlowell follow the above steps to test for go. When we reached step 6:

echo (az vm run-command invoke --ids /subscriptions/{subscription}/resourceGroups/{resource group}/providers/Microsoft.Compute/virtualMachines/{VM name} --command-id RunShellScript --scripts "cd /tmp/azure-sdk-for-go/sdk/azidentity && export AZURE_VAULT_URL=https://kvcpname0.vault.azure.net && go test -run TestManagedIdentityCredential_GetTokenInVMLive" -o json --query "value[0].message")

Add command go get github.com/Azure/azure-sdk-for-go/sdk/azidentity@v1.3.0-beta.5.0.20230425151240-402376456ba8 to install identity version v1.3.0-beta.5.0.20230425151240-402376456ba8 before command go test -run TestManagedIdentityCredential_GetTokenInVMLive . get error as follow:

Through investigation, after clone the code, switch to azure-sdk-for-go/sdk/samples/azidentity/manual-tests/managed-identity/general and run go run main.go. It works normally. For more details, please refer to Azure Arc testing.

For Azure VM MSI testing for go, can we refer to Azure Arc testing to update testing instructions?

[github-actions]

Hi @jianghaolu, we deeply appreciate your input into this project. Regrettably, this issue has remained inactive for over 2 years, leading us to the decision to close it. We've implemented this policy to maintain the relevance of our issue queue and facilitate easier navigation for new contributors. If you still believe this topic requires attention, please feel free to create a new issue, referencing this one. Thank you for your understanding and ongoing support.

[v-xuto]

@joshfree This issue has been closed. Do we need to continue testing this Automate VM MSI testing?