Makefile rule "deploy-shared-service" fails during firewall deployment

[migldasilva]

I've been working lately days on deploying shared services, and couldn't sort a problem regarding the tre-shared-service-firewall.

Describe the bug
The Makefile on the root of the repo has the following rule:

deploy-shared-service:
	@# NOTE: ACR_NAME below comes from the env files, so needs the double '$$'. Others are set on command execution and don't
	$(call target_title, "Deploying ${DIR} shared service") \
	&& . ${MAKEFILE_DIR}/devops/scripts/check_dependencies.sh porter,env,auth \
	&& . ${MAKEFILE_DIR}/devops/scripts/get_access_token.sh \
	&& cd ${DIR} \
	&& ${MAKEFILE_DIR}/devops/scripts/deploy_shared_service.sh --insecure --tre_url "$${TRE_URL:-https://$${TRE_ID}.$${LOCATION}.cloudapp.azure.com}" $${PROPS}

Everything goes fine until the script deploy_shared_service.sh is called. It means, for instance, that the access token is correctly retrieved. Besides that, after expecting the deploy_shared_service.sh I could conclude that the Swagger UI is reachable, and the POST call to /api/shared-services is sent.

On the command line, I can follow the process without problems:

Waiting for deployment of tre-shared-service-firewall to finish... (current status: awaiting_deployment)
Waiting for deployment of tre-shared-service-firewall to finish... (current status: awaiting_deployment)
Waiting for deployment of tre-shared-service-firewall to finish... (current status: awaiting_deployment)
Waiting for deployment of tre-shared-service-firewall to finish... (current status: deploying)
Waiting for deployment of tre-shared-service-firewall to finish... (current status: deploying)
Waiting for deployment of tre-shared-service-firewall to finish... (current status: deploying)
Waiting for deployment of tre-shared-service-firewall to finish... (current status: deploying)
Waiting for deployment of tre-shared-service-firewall to finish... (current status: deploying)
Waiting for deployment of tre-shared-service-firewall to finish... (current status: deploying)
Waiting for deployment of tre-shared-service-firewall to finish... (current status: deploying)
Waiting for deployment of tre-shared-service-firewall to finish... (current status: deploying)
Waiting for deployment of tre-shared-service-firewall to finish... (current status: deploying)
Waiting for deployment of tre-shared-service-firewall to finish... (current status: deploying)
Waiting for deployment of tre-shared-service-firewall to finish... (current status: deploying)
Waiting for deployment of tre-shared-service-firewall to finish... (current status: deploying)
Waiting for deployment of tre-shared-service-firewall to finish... (current status: deploying)
Waiting for deployment of tre-shared-service-firewall to finish... (current status: deploying)
Waiting for deployment of tre-shared-service-firewall to finish... (current status: deployment_failed)
Failed to await operation cef4de1d-d995-4dbb-8d04-123456789abc (status is deployment_failed). Please check resource processor logs
make[1]: *** [/home/vscode/AzureTRE/Makefile:268: deploy-shared-service] Error 1
make[1]: Leaving directory '/workspaces/cprd-tre'
make: *** [/home/vscode/AzureTRE/Makefile:275: firewall-install] Error 2

The same is valid when using the Swagger UI. Swagger output is more verbose, but in both cases they are consistent and show the same information. To dig a little bit deeper, I enabled debugging in deploy-shared-service.sh script, and end up getting this message (it has been sanitized):

{"operation":{"id":"cef4de1d-d995-4dbb-8d04-123456789abc","resourceId":"5d936479-5e36-4752-b2af-123456789abc","resourcePath":"/shared-services/5d936479-5e36-4752-b2af-123456789abc","resourceVersion":0,"status":"deployment_failed","action":"install","message":"5d936479-5e36-4752-b2af-123456789abc: Error context message = Error: could not load credentials: 4 errors occurred: \t* unable to resolve credential azure.azure_client_id from env ARM_CLIENT_ID: could not connect to the secrets.azure.keyvault plugin: Incompatible API version with plugin. Plugin version: 2, Client versions: [1]: {\"@level\":\"debug\",\"@message\":\"plugin address\",\"@timestamp\":\"2022-10-25T11:45:53.560023Z\",\"address\":\"/tmp/plugin2614558627\",\"network\":\"unix\"} \t* unable to resolve credential azure.azure_client_secret from env ARM_CLIENT_SECRET: could not connect to the secrets.azure.keyvault plugin: Incompatible API version with plugin. Plugin version: 2, Client versions: [1]: {\"@level\":\"debug\",\"@message\":\"plugin address\",\"@timestamp\":\"2022-10-25T11:45:53.571301Z\",\"address\":\"/tmp/plugin2286842278\",\"network\":\"unix\"} \t* unable to resolve credential azure.azure_subscription_id from env ARM_SUBSCRIPTION_ID: could not connect to the secrets.azure.keyvault plugin: Incompatible API version with plugin. Plugin version: 2, Client versions: [1]: {\"@level\":\"debug\",\"@message\":\"plugin address\",\"@timestamp\":\"2022-10-25T11:45:53.579289Z\",\"address\":\"/tmp/plugin476324807\",\"network\":\"unix\"} \t* unable to resolve credential azure.azure_tenant_id from env ARM_TENANT_ID: could not connect to the secrets.azure.keyvault plugin: Incompatible API version with plugin. Plugin version: 2, Client versions: [1]: {\"@level\":\"debug\",\"@message\":\"plugin address\",\"@timestamp\":\"2022-10-25T11:45:53.588893Z\",\"address\":\"/tmp/plugin3907949306\",\"network\":\"unix\"} az login --identity -u d46f3ca3-a1f1-47ab-8243-123456789abc && az acr login --name acrtredevmgmt && porter install \"5d936479-5e36-4752-b2af-123456789abc\" --reference acrtredevmgmt.azurecr.io/tre-shared-service-firewall:v0.4.4 --param arm_use_msi=\"true\" --param id=\"5d936479-5e36-4752-b2af-123456789abc\" --param tfstate_container_name=\"tfstate\" --param tfstate_resource_group_name=\"rg-tredev-mgmt\" --param tfstate_storage_account_name=\"tredevmgmt\" --param tre_id=\"tredev\" --allow-docker-host-access --force --cred ./vmss_porter/arm_auth_local_debugging.json --cred ./vmss_porter/aad_auth.json","createdWhen":1666698347.200221,"updatedWhen":1666698370.64461,"user":{"id":"caa2cda6-b64f-459f-9815-07c7a8750795","name":"","email":"","roles":["TREAdmin"],"roleAssignments":[]},"steps":[{"stepId":"main","stepTitle":"Main step for 5d936479-5e36-4752-b2af-123456789abc","resourceId":"5d936479-5e36-4752-b2af-123456789abc","resourceTemplateName":"tre-shared-service-firewall","resourceType":"shared-service","resourceAction":"install","status":"deployment_failed","message":"5d936479-5e36-4752-b2af-123456789abc: Error context message = Error: could not load credentials: 4 errors occurred: \t* unable to resolve credential azure.azure_client_id from env ARM_CLIENT_ID: could not connect to the secrets.azure.keyvault plugin: Incompatible API version with plugin. Plugin version: 2, Client versions: [1]: {\"@level\":\"debug\",\"@message\":\"plugin address\",\"@timestamp\":\"2022-10-25T11:45:53.560023Z\",\"address\":\"/tmp/plugin2614558627\",\"network\":\"unix\"} \t* unable to resolve credential azure.azure_client_secret from env ARM_CLIENT_SECRET: could not connect to the secrets.azure.keyvault plugin: Incompatible API version with plugin. Plugin version: 2, Client versions: [1]: {\"@level\":\"debug\",\"@message\":\"plugin address\",\"@timestamp\":\"2022-10-25T11:45:53.571301Z\",\"address\":\"/tmp/plugin2286842278\",\"network\":\"unix\"} \t* unable to resolve credential azure.azure_subscription_id from env ARM_SUBSCRIPTION_ID: could not connect to the secrets.azure.keyvault plugin: Incompatible API version with plugin. Plugin version: 2, Client versions: [1]: {\"@level\":\"debug\",\"@message\":\"plugin address\",\"@timestamp\":\"2022-10-25T11:45:53.579289Z\",\"address\":\"/tmp/plugin476324807\",\"network\":\"unix\"} \t* unable to resolve credential azure.azure_tenant_id from env ARM_TENANT_ID: could not connect to the secrets.azure.keyvault plugin: Incompatible API version with plugin. Plugin version: 2, Client versions: [1]: {\"@level\":\"debug\",\"@message\":\"plugin address\",\"@timestamp\":\"2022-10-25T11:45:53.588893Z\",\"address\":\"/tmp/plugin3907949306\",\"network\":\"unix\"} az login --identity -u d46f3ca3-a1f1-47ab-8243-f68db508baba && az acr login --name acrtredevmgmt && porter install \"5d936479-5e36-4752-b2af-123456789abc\" --reference acrtredevmgmt.azurecr.io/tre-shared-service-firewall:v0.4.4 --param arm_use_msi=\"true\" --param id=\"5d936479-5e36-4752-b2af-123456789abc\" --param tfstate_container_name=\"tfstate\" --param tfstate_resource_group_name=\"rg-tredev-mgmt\" --param tfstate_storage_account_name=\"tredevmgmt\" --param tre_id=\"tredev\" --allow-docker-host-access --force --cred ./vmss_porter/arm_auth_local_debugging.json --cred ./vmss_porter/aad_auth.json","updatedWhen":1666698370.644598}]}}'

Please observe that the firewall version is 0.4.4. It means that I made a change in the file porter.yaml, in order to increase this version number. The original one was 0.4.3, and even tough the deployment failed.

This message was really cryptic: unable to resolve credential azure.azure_tenant_id from env ARM_TENANT_ID: could not connect to the secrets.azure.keyvault plugin: Incompatible API version with plugin. Plugin version: 2, Client versions: [1]:

Which API is this one?

What I have tried so far:

1. Provide hardcoded values for the variables ARM_SUBSCRIPTION_ID, ARM_CLIENT_SECRET, ARM_CLIENT_ID and ARM_TENANT_ID. Actually, the set-up process includes setting ARM_SUBSCRIPTION_ID variable in the file devops/.env.
2. Use Swagger UI for launching the deployment. It means, instead of using make command.

Thank you very much!

Steps to reproduce

1. Set-up TRE as explained in the official site
2. Launch VS Code and use a devcontainer
3. Run make all

[marrobi]

This looks like - #2761 - a change in a dependency, we have now pinned this.

This has been fixed in release v0.6.0.

Let us know if you still encounter issues after upgrading - please check breaking changes in the change log.