-
Notifications
You must be signed in to change notification settings - Fork 302
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Automate Pod Identity testing #814
Comments
@chlowel, it looks like you've got most of this figured out, what assistance do you need from the engineering system team? |
I want to replace all the manual steps above with an automated live test. I need a pipeline which builds the artifacts (in particular, Docker images), creates and destroys the Azure resources, runs the test script, and reports results. |
For JavaEverything is the same prior to Build Images, with the ACR, AKS, Key Vault, managed identity created, and the managed identity is given secret permissions to the key vault. Additionally, add az keyvault set-policy -n $KEY_VAULT_NAME --object-id $MANAGED_IDENTITY_PRINCIPAL_ID --secret-permissions set delete get And load the key vault with a secret called az keyvault secret set --vault-name $KEY_VAULT_NAME -n secret --value "The secret value" Build ImagesThe test application must be packaged as a Docker image before deployment. 1. authenticate to ACRGet credentials: az acr credential show -n $ACR_NAME -o table Authenticate with the "USERNAME" and either "PASSWORD": az acr login -n $ACR_NAME -u <USERNAME> -p <PASSWORD> 2. Create DockerfileCreate a new directory, e.g. FROM alpine/git as clone
RUN git clone https://github.com/Azure/azure-sdk-for-java --single-branch --depth 1 /azure-sdk-for-java
FROM maven:3-jdk-8
COPY --from=clone /azure-sdk-for-java /azure-sdk-for-java
WORKDIR /azure-sdk-for-java
RUN mvn clean install -Dgpg.skip -DskipTests -f eng/code-quality-reports/pom.xml
RUN mvn clean install -Dgpg.skip -DskipTests -f common/perf-test-core/pom.xml
RUN mvn clean install -Dgpg.skip -DskipTests -Dmaven.javadoc.skip=true -f sdk/core/azure-core/pom.xml
RUN mvn clean install -Dgpg.skip -DskipTests -Dmaven.javadoc.skip=true -f sdk/core/azure-core-test/pom.xml
RUN mvn clean install -Dgpg.skip -DskipTests -Dmaven.javadoc.skip=true -f sdk/core/azure-core-http-netty/pom.xml
RUN mvn clean install -Dgpg.skip -DskipTests -Dmaven.javadoc.skip=true -f sdk/identity/azure-identity/pom.xml
RUN mvn clean install -Dgpg.skip -DskipTests -Dmaven.javadoc.skip=true -f sdk/keyvault/azure-security-keyvault-secrets/pom.xml
RUN mvn clean install -Dgpg.skip -DskipTests -Dmaven.javadoc.skip=true -f sdk/keyvault/azure-security-keyvault-keys/pom.xml
RUN mvn clean install -Dgpg.skip -DskipTests -Dmaven.javadoc.skip=true -f sdk/keyvault/azure-security-keyvault-certificates/pom.xml
CMD [ "mvn", "test", "-Dtest=ManagedIdentityCredentialLiveTest#testMSIEndpoint*", "-f", "sdk/e2e/pom.xml", "-Dgpg.skip", "-am", "-DfailIfNoTests=false" ] 3. Build image and push to the container registryexport REPOSITORY=$ACR_NAME.azurecr.io IMAGE_NAME=test-pod-identity
docker build -t $REPOSITORY/$IMAGE_NAME ./pod-identity-test
docker push $REPOSITORY/$IMAGE_NAME Run the test1. install kubectlaz aks install-cli 2. authenticate kubectlaz aks get-credentials -g $RESOURCE_GROUP -n $AKS_NAME 3. Create
|
For GolangFollow all of the same steps as JAVA, the only change is in the Dockerfile in step 2 of Build Images. For Go the content of the Dockerfile should be:
Expected output if the test succeeds: PASS |
Hi @chlowell , Same problem as Automate VM MSI testing. we need to create a non-soft-delete KV.
|
Yes, soft delete was disabled by default when I wrote these instructions but is now enabled by default for new vaults. The right solution is to change the tests so this doesn't matter. I've opened Azure/azure-sdk-for-python#10879 to do that. |
For JavaScript: Testing azure-identity in Azure Kubernetesprerequisite tools
Azure resourcesThis test requires instances of these Azure resources:
The rest of this section is a walkthrough of deploying these resources. set environment variables to simplify copy-pasting
resource groupaz group create -n $RESOURCE_GROUP --location westus2 managed identityCreate the managed identity: az identity create -g $RESOURCE_GROUP -n $MANAGED_IDENTITY_NAME Save its $MANAGED_IDENTITY_CLIENT_ID=az identity show -g $RESOURCE_GROUP -n $MANAGED_IDENTITY_NAME --query clientId -o tsv
$MANAGED_IDENTITY_ID=az identity show -g $RESOURCE_GROUP -n $MANAGED_IDENTITY_NAME --query id -o tsv
$MANAGED_IDENTITY_PRINCIPAL_ID=az identity show -g $RESOURCE_GROUP -n $MANAGED_IDENTITY_NAME --query principalId -o tsv Key VaultCreate the Vault: az keyvault create -g $RESOURCE_GROUP -n $KEY_VAULT_NAME --sku standard Add an access policy for the managed identity: az keyvault set-policy -n $KEY_VAULT_NAME --object-id $MANAGED_IDENTITY_PRINCIPAL_ID --secret-permissions set delete container registryaz acr create -g $RESOURCE_GROUP -n $ACR_NAME --admin-enabled --sku basic KubernetesDeploy the cluster (this will take several minutes): az aks create -g $RESOURCE_GROUP -n $AKS_NAME --generate-ssh-keys --node-count 1 --disable-rbac --attach-acr $ACR_NAME Grant the cluster's service principal permission to use the managed identity: az role assignment create --role "Managed Identity Operator" --assignee $(az aks show -g $RESOURCE_GROUP -n $AKS_NAME --query servicePrincipalProfile.clientId -o tsv) --scope $MANAGED_IDENTITY_ID build imagesThe test application must be packaged as a Docker image before deployment. authenticate to ACRaz acr login -n $ACR_NAME acquire the test codegit clone https://github.com/Azure/azure-sdk-for-js/ --branch master --single-branch --depth 1 The rest of this section assumes this working directory: cd azure-sdk-for-js/sdk/identity/identity/test/manual-integration/kubernetes build images and push them to the container registrySet environment variables: $REPOSITORY="$($ACR_NAME).azurecr.io"
$IMAGE_NAME="test-pod-identity"
$NODE_VERSION=10 Build an image: docker build --no-cache --build-arg NODE_VERSION=$NODE_VERSION -t "$($REPOSITORY)/$($IMAGE_NAME):$($NODE_VERSION)" . Push it to ACR: docker push "$($REPOSITORY)/$($IMAGE_NAME):$($NODE_VERSION)" run the testinstall kubectlaz aks install-cli authenticate kubectl and helmaz aks get-credentials -g $RESOURCE_GROUP -n $AKS_NAME install tillerhelm init --wait run the test scriptnpm install tsc -p . node ./run_test.js --client-id $MANAGED_IDENTITY_CLIENT_ID --resource-id $MANAGED_IDENTITY_ID --vault-url "https://$($KEY_VAULT_NAME).vault.azure.net" --repository $REPOSITORY --image-name $IMAGE_NAME --image-tag $NODE_VERSION verify success
delete Azure resourcesaz group delete -n $RESOURCE_GROUP -y --no-wait |
Hi @jonathandturner, I am following above steps to do JS E2E test. Below error occurred after running Could you please help to update the test steps to resolve the error? |
@catalinaperalta -I am following the instrucations to do go E2E test. Below error occurred when executing last command.
The test passed last month but fials this time. Could you help to take a look to see if there is anything changed? |
@chlowell - You might miss adding the list permission to key vault. |
@jianghaolu -We am following the instrucations to do Java E2E test. Below error occurred after running all commands. We have tried for many times, but still the same error. Could you help to check what is causing this issue? |
Hi @jianghaolu @catalinaperalta - following E2E test It is related to the vmss, so I try to access 169.254.169.254 using az cli(az vmss run-command invoke). Container initialization failed, Could you help to check what is causing this issue? |
Hi @jianghaolu @catalinaperalta. I am following above steps to do JS E2E test. Below error occurred after running I tried to update
|
@sadasant @KarishmaGhiya for JS issues |
Give us some time to sync with this issue. |
@chlowell thank you, Charles! @v-jiaodi This happens because we forgot to specify the typescript version we used to compile this project originally. I’ve made a pull request in which I’m fixing the TypeScript version to the latest one available, and I’m also fixing the types. In the future, this specific problem will not happen. Thank you for letting us know! Here’s the PR: Azure/azure-sdk-for-js#17910 |
@sadasant This issue has not been completely solved,the following is the error message: |
I will investigate at first hour. Thank you! |
I just saw the issue. It was my bad, I didn’t re-try building after I addressed the feedback I got. Here’s the fix: Azure/azure-sdk-for-js#18103 |
I use the branch test you mentioned, and this issue no longer exists. |
Thank you! |
Hi @chlowell. The folder |
Given that Pod Identity is fully deprecated and won't ship another release, and we support it with our code for IMDS managed identity--which we continue to test--I don't think we need to continue testing Pod Identity separately. So, I'll close this issue. @christothes please reopen or comment if you disagree |
Pod Identity is now deprecated
https://github.com/Azure/aad-pod-identity#aad-pod-identity-deprecated
use workload identity instead
We need to automate testing our identity libraries' managed identity implementations with pod identity (AKS).
Below are instructions for manually testing Python's implementation (as seen in the repo):
Testing managed identity in Azure Kubernetes Service
prerequisite tools
Azure resources
This test requires instances of these Azure resources:
The rest of this section is a walkthrough of deploying these resources.
set environment variables to simplify copy-pasting
resource group
az group create -n $RESOURCE_GROUP --location westus2
managed identity
Create the managed identity:
Save its
clientId
,id
(ARM URI), andprincipalId
(object ID) for later:Key Vault
Create the Vault:
Add an access policy for the managed identity:
container registry
Kubernetes
Deploy the cluster (this will take several minutes):
Grant the cluster's service principal permission to use the managed identity:
build images
The test application must be packaged as a Docker image before deployment.
Test runs must include Python 2 and 3, so two images are required.
authenticate to ACR
az acr login -n $ACR_NAME
acquire the test code
The rest of this section assumes this working directory:
cd azure-sdk-for-python/sdk/identity/azure-identity/tests
build images and push them to the container registry
Set environment variables:
Build an image:
Push it to ACR:
Then set
PYTHON_VERSION
to the latest 3.x (3.8 at time of writing) and run theabove
docker build
anddocker push
commands again. (It's safe--and faster--to omit
--no-cache
fromdocker build
the second time.)run the test
install kubectl
authenticate kubectl and helm
install tiller
run the test script
Twice. Once with
PYTHON_VERSION=2.7
, once withPYTHON_VERSION=3.x
(replacing x with the latest Python 3 minor version):
delete Azure resources
az group delete -n $RESOURCE_GROUP -y --no-wait
The text was updated successfully, but these errors were encountered: