Unable to deploy bundle due to incompatible API version

[migldasilva]

I've being using AzureTRE solution and lately I'm not unable to deploy one of the components. The error messages are shown below:

unable to resolve credential azure.azure_subscription_id from env ARM_SUBSCRIPTION_ID: could not connect to the secrets.azure.keyvault plugin: Incompatible API version with plugin. Plugin version: 2, Client versions: [1]

The same message repeats for 3 more variables: ARM_CLIENT_ID, ARM_CLIENT_SECRET and ARM_TENANT_ID.

Following Azure Plugins' documentation I found the following variables: AZURE_CLIENT_ID, AZURE_CLIENT_SECRET and AZURE_TENANT_ID. It seams that they correspond to the ones listed above.

I already raised a ticket with AzureTRE team, giving that the error message does not make clear if it's due to the Azure Plugins, or something in Azure Key Vault.

So far I have tested all the combinations of the following versions:

1. Porter v0.38.12 and v0.38.13
2. Azure Pluguins: v1.0.1 and v0.11.0

Thank you very much.

[carolynvs]

Hi @migldasilva, apologies for the confusion around compatibility. I believe Azure TRE is still using Porter v0.38.13, which is not compatible with the v1 releases of the azure plugin.

When you have both Porter v0.38.13 and Azure plugin v0.11.2 installed, can you run the following command to verify and then reply back with the error message received when you try to run a bundle? I am hoping that once we are sure that the versions used are compatible that the bundle will run successfully.

porter version
porter plugins list -o yaml

[migldasilva]

Hello @carolynvs , thank you so much for the support. Actually, the newest version of AzureTRE (0.6.0) is using porter v0.38.13. However, the version I'm deploying (it's, version 0.4.3) is using porter v0.38.12.

Here is the info you asked for:

$ porter version
porter v0.38.12 (26b4ba0e)
$ porter plugins list -o yaml
- name: azure
  versioninfo:
    version: v0.11.2
    commit: 894e361
    author: Porter Authors
  implementations:
    - type: storage
      name: blob
    - type: secrets
      name: keyvault

I have just prepared everything to start trying AzureTRE using porter v0.38.12 and Azure Plugins v0.11.2.

I'll keep you posted.

[migldasilva]

Unfortunately, no luck. The same error still shows up even though Porter v0.38.13 and Azure Plugins v0.11.2 are being used.

unable to resolve credential azure.azure_client_id from env ARM_CLIENT_ID: could not connect to the secrets.azure.keyvault plugin: Incompatible API version with plugin. Plugin version: 2, Client versions: [1]: {\"@level\":\"debug\",\"@message\":\"plugin address\",\"@timestamp\":\"2022-10-26T07:49:57.389122Z\",\"address\":\"/tmp/plugin706193066\",\"network\":\"unix\"}

Actually, something got my attention now. After the Plugin and Client versions are printed, a JSON object is also printed. Which component is generating this message? I may be wrong, but it really looks like Terraform's debug output.

Besides, another question, which is "problematic API" is this case? I mean, there seems to be an incompatibility between an API and the Azure Plugins. Is this an Azure Key Vault API?

Thanks.

[carolynvs]

You're right that this looks like an error message from terraform, because we are using the same plugin framework (hashicorp/go-plugin) to communicate between Porter and Porter's plugins. The error indicates that Porter is on the secrets protocol version 1, but the plugin itself is on the newer version 2 (which is only compatible with Porter v1).

Do you have more information about how this is being run? Based on the output of the commands for porter plugins list, you should be using the older azure plugin and should not be getting this error. I'm wondering if there's something more complex in the actual environment that is causing the wrong version of the Azure plugin to be used. Perhaps you are using a custom build of the Azure plugin or the environment where the bundle is being run isn't installing a specific version of the plugin, and therefore is picking up the most recent version (which just came out last week, previously installing the "latest" version would have picked up v0.11.2).

If the bundle isn't public, or it would work better to do a screenshare, you can reach me at (removed) and we can set up a call to troubleshoot.

[migldasilva]

Thanks for all the explanation. It's more than clear now.

The environment I'm trying to run is using the regular artifacts available on the official AzureTRE repo. There's a Makefile available, and for setting everything up, it's supposed to be enough just to run make all.

The whole process takes from 1 to 2 hours to complete. I can follow the status on command line, or indirectly through Azure Web UI (for instance, I can see objects being added to the corresponding Azure subscription). Close to the end of the process, a step fails.

I've been debugging the Makefile rules that create the failing bundle. Everything seems fine. Such rule is executed inside a DevContainer attached to VisualStudio Code. During the container creation, the following artifacts are installed:

1. Porter
2. Terrarform Porter Mixin (v1.0.0-rc.1)
3. Az Porter Mixin (v0.7.3)
4. Docker Porter Mixin (v0.3.0)
5. Azure Plugins (v0.11.2)
6. Exec Porter Mixin (latest)

Originally, all these artifacts were installed in the latest version. However, I have pinned them all (it means that I had to change a Dockerfile and the Bash script installing these components). Porter is download from https://cdn.porter.sh, and plunins/mixins are installed using Porter (i.e., porter mixin install <MIXIN> --version=<PINNED_VERSION>, and porter plugin install <PLUGIN> --version=<PINNED_VERSION>).

When this DevContainer is running I can execute porter in order to check all the versions. They are indeed the ones I selected for insatllation. And as far as I could track, Porter, plugins and mixins are not changed/upgraded/installed during bundle generation, or when any of the available Makefile rules are executed.

I couldn't figure out how a non-compatible version is being dragged into the scenario!

Thanks!

[migldasilva]

I could dig down to the root!

Essentially, AzureTRE is packaged so that Porter and its plugins are installed multiple times, in different containers. Such containers are used for installing the bundles.

Thus, after pinning the desired versions in such containers, I can confirm that Porter version 0.38.12 and Azure Plugins v0.11.2 are compatible.

Thank you all for the assistance.