Vault provider does not tolerate token provided

[dimitryvoronov]

Steps to reproduce

1. Create vault provider and secret for namespace serviceaccount

	// vault provider
	if !namespacesConfig.VaultDisabled {
		vaultProvider, err = vault.NewProvider(ctx, "vault-provider", &vault.ProviderArgs{
			Address: pulumi.String(namespacesConfig.VaultConfig.Url),
			Token:   pulumi.String(namespacesConfig.VaultConfig.Token),
		})
		if err != nil {
			return nsNamesToNsResources, err
		}
	}
		if !namespacesConfig.VaultDisabled {
			_, err = generic.NewSecret(ctx, fmt.Sprintf("deployer-vault-secret-%s", ns), &generic.SecretArgs{
				DisableRead: pulumi.Bool(true),
				DataJson: pulumi.Sprintf(`{"ACCESS_KEY_ID": "%s", "ACCESS_SECRET_KEY": "%s"}`,
					deployerAccessKey.ID(), deployerAccessKey.Secret),
				Path: pulumi.String("environment/aws/" + ns),
			}, pulumi.Provider(vaultProvider))
			if err != nil {
				return nsNamesToNsResources, err
			}
		}

Please notice that we optionally define vault token and addr.
2. Run pulumi up or pulumi update to get info about vault objects in stack config
3. Examine stack export for vault token

bash> pulumi stack export --file=myexport.stack.config.json 
          {
                "urn": "urn:pulumi:cl03::create-cluster::pulumi:providers:vault::vault-provider",
                "custom": true,
                "id": "741c6942-97b4-4240-a617-a2ee2ab12465",
                "type": "pulumi:providers:vault",
                "inputs": {
                    "address": "https://magic-vault.example.com:8082",
                    "caCertDir": "",
                    "caCertFile": "",
                    "maxLeaseTtlSeconds": 1200,
                    "maxRetries": 2,
                    "namespace": "",
                    "skipTlsVerify": false,
                    "token": "s.SWv44uu9escNeEzzglERavHv",
                    "tokenName": ""
                },

Expected:
We expect that vault token provider either reads definition on provider level of Address and Token OR
VAULT_ADDR and VAULT_TOKEN environment variables.
Actual:
After the first run token which stored in pulumi stack is not updated anymore.It was working fine for a default period of time o the token which is 32 days, then all our jobs started to fail with
URL: GET https://magic-vault.example.com:8082/v1/auth/token/lookup-self
Code: 403. Errors:

* permission denied

After investigation we found that old, expired token is stored on pulumi stack config and provider always reads it.

We checked this behavior on couple versions, the latest and 2.4.1(used originally)
Then we explicitly defined Address and Token on provider level and set both via config settings and that did not help.
Each time we run pulumi up or pulumi update vault provider still reads token ONLY from pulumi stack (stored on S3 bucket)

[leezen]

@dimitryvoronov Are you able to provide a repro case? We would expect the values in the program to be used. Have you confirmed that that block where you're providing values gets used and that the provider instance that consumes those values is provided to the resource in question?

[dimitryvoronov]

Hello @leezen
We define a few values on pulumi config level, for example for part which creates a vault provider we define, addr and token.
Only one pulumi vault provider is created when the project runs, but for each namespace we create a separate secret which stores on vault.

config:
  create-cluster:k8snamespace:
    namespaces:
      - namespace-01
      - namespace-02
    vaultConfig:
      token: s.pzzzzzzzzzzzzzzzzzzzzzzz
      url: https://magic-vault.example.com:8082

So, provider as I understand should use the values provided to it, but it's not, and even the token we provide via value does not appear in stack config. Do you know if it should be written in stack config once you defined it in code?

[dmattia]

@leezen This is not trivial to create a repro case for as it requires having a token in the statefile become stale, but I did just run into this.

The workarounds that I found to work are:

• Do not pass in a token, but use authLogins in the provider config. This is fine, but does not work with AWS SSO for AWS IAM logins
• Changing the name of the provider, which will ensure the old value in the statefile is not used

These are both still unideal, but provided me with enough of a workaround for now

[dmattia]

Might this have to do with passing --refresh?

[Igopin]

@leezen @dmattia I think I can provide a repro case:

1. Let's consider such code (I'm using s3 bucket for state storage):

package main

import (
        "github.com/pulumi/pulumi-vault/sdk/v4/go/vault"
        "github.com/pulumi/pulumi-vault/sdk/v4/go/vault/generic"
        "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
        "github.com/pulumi/pulumi/sdk/v3/go/pulumi/config"
)

func main() {
        pulumi.Run(func(ctx *pulumi.Context) error {
                cfg := config.New(ctx, "")

                vaultProvider, err := vault.NewProvider(ctx, "vault-provider", &vault.ProviderArgs{
                        Address: pulumi.String(cfg.Require("vault_address")),
                        Token:   pulumi.String(cfg.Require("vault_token")),
                })

                if err != nil {
                        return err
                }

                secret, err := generic.NewSecret(
                        ctx,
                        "test",
                        &generic.SecretArgs{
                                DataJson:    pulumi.String(`{"password": "p@assword"}`),
                                Path:        pulumi.String("path/to/secret"),
                        },
                        pulumi.Provider(vaultProvider),
                )
  
                return err
        })
}

2. Generate vault token and add it to the pulumi config: pulumi config set --secret vault_token <token>
3. Run pulumi up to create a secret, everything should be okay
4. To simulate token exploration just revoke the token (or you can create token with short ttl): vault token revoke <token>
5. Provide a new valid vault token to the pulumi config: pulumi config set --secret vault_token <token>
6. Try to run whether pulumi refresh or pulumi destroy and you will get an error:

...
Diagnostics:
  pulumi:pulumi:Stack (test-expired-token):
    error: preview failed
 
  vault:generic:Secret (test):
    error: Preview failed: Error making API request.
    
    URL: GET https://host-of-the-vault-instance/v1/auth/token/lookup-self
    Code: 403. Errors:
    
    * permission denied
...

As @dimitryvoronov mentioned it happens due to expired token stored into statefile and only one solution I found is to update vault provider before any of those actions. Something like:

 VAULT_PROVIDER_URN=$(pulumi stack export | jq -r '[.deployment.resources[] | select(.urn|test("::pulumi:providers:vault::")) | .urn][0]
pulumi up --target $VAULT_PRVODER_URN

But it looks like a hack (and actualy it is) and I'd prefer to find better approach.

[AMKamel]

I think I am having the same issue, using typescript.

const sesDomainVaultCredentials = new vault.generic.Secret(`${tenant}-${deploymentEnvironment}-${fqdn}-ses-domain`, {
        dataJson: JSON.stringify(
            {
                "access_key_id": sesDomain.awsSesAccessKeyId,
                "secret_access_key": sesDomain.awsSesSecretAccessKey,
                "smtp_user": sesDomain.awsSesSmtpUser,
                "smtp_password": sesDomain.awsSesSmtpPassword,
                "smtp_host": sesDomain.awsSesSmtpHost
            },
        ),
        path: `secret/application/ses`,
    }
)

when running pulumi destroy I got an error, I tried to delete secret with vault cli and it succeeded, so I guessed that the provider is using the old vault token and it didn't read the new one provided via VAULT_TOKEN or pulumi config, I tried both.

Error:

  vault:generic:Secret ():
    error: deleting urn:pulumi:dev::: 1 error occurred:
    	* error deleting "" from Vault: "Error making API request.\n\nURL: DELETE \nCode: 403. Errors:\n\n* 1 error occurred:\n\t* permission denied\n\n"

[AMKamel]

UPDATE: it didn't work, again deletion failed!
There is an up normal behavior with this provider it seems like a bug to me.

destroying everything in the stack first then using following provider options solved it for me for now, using userpass auth

const vaultAppRoleLoginConfig: vault.types.output.config.AuthLogins = {
    path: `/auth/userpass/login/${process.env.VAULT_USERNAME}`,
    namespace: vault.config.namespace,
    parameters: {
        password: process.env.VAULT_PASSWORD!
    }
};

const vaultProvider = new vault.Provider("userpass", {
    token: "you literally have to write some crap here, otherwise it will not work",
    address: vault.config.address!,
    namespace: vault.config.namespace,
    authLogins: [vaultAppRoleLoginConfig]
})

[CmdrSharp]

This is indeed an issue. In our case, we fetch login details, do an explicit login and pass that to the provider. It works the first time, until that token expires - at that point in time, the stack value is the only thing Pulumi cares about.

    const approle = await vault.generic.getSecret({
      path: "kv/pulumi"
    });

    const login = new vault.approle.AuthBackendLogin("login", {
      backend: "approle",
      roleId: approle.data.roleId,
      secretId: approle.data.secretId,
    });

    this._provider = new vault.Provider("namespace-provider", {
      address: this.vaultProviderConfig.require("address"),
      token: login.clientToken,
      namespace: `${this.config.require("root")}/${this.name}`,
    });

Not so sure there's an easy fix though. Here's how I solved it for using an approle:

    this._provider = new vault.Provider("namespace-provider", {
      address: this.vaultProviderConfig.require("address"),
      token: "-",  // Required parameter, but not used.
      authLogin: {
        path: `auth/approle/login`,
        namespace: this.config.require("root"),
        parameters: {
          role_id: approle.data.roleId,
          secret_id: approle.data.secretId
        }
      },
      namespace: `${this.config.require("root")}/${this.name}`,
    });

[corest]

Also run into this issue today.
They way I fixed it for now was this:

1. Export stack state

pulumi stack export --stack <stack name>file stack.json

2. Open stack.json and find vault provider resource

It will be something like

...
            {
                "urn": "urn:pulumi:dev::Application::pulumi:providers:vault::default",
                "custom": true,
                "id": "17842636-8c43-402b-8ef0-260b16135338",
                "type": "pulumi:providers:vault",
                "inputs": {
                    "address": "<addr>",
                    "token": {
                        "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
                        "ciphertext": "v1:pUlh7L4tcbE+ypdd:rNap1TsecretEIZmhNRs="
                    },
                    "version": "5.7.2"
                },
                "outputs": {
                    "address": "<addr>",
                    "token": {
                        "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
                        "ciphertext": "v1:pUlh7L4tcbE+ypdd:rNap1TnEsecretGeBxdu6FEIZmhNRs="
                    },
                    "version": "5.7.2"
                }
            },
...

Delete token key from resource input/output and save file

...
            {
                "urn": "urn:pulumi:dev::Application::pulumi:providers:vault::default",
                "custom": true,
                "id": "17842636-8c43-402b-8ef0-260b16135338",
                "type": "pulumi:providers:vault",
                "inputs": {
                    "address": "<addr>",
                    "version": "5.7.2"
                },
                "outputs": {
                    "address": "<addr>",
                    "version": "5.7.2"
                }
            },
...

3. Import stack back

pulumi stack import --stack <stack name>-file stack.json

4. update/refresh pulumi

[asvinours]

This issue is still happening, and other than the fact that this makes using the pulumi-vault provider very difficult, it's also the only plain-text secret saved in the pulumi checkpoint file.

I think if the bug is not gonna be fixed, at minima a big red warning should be added to the provider documentation page to let people know that this provider saves credentials in clear-text in the checkpoint file as this is not how pulumi usually works.

[gawsoftpl]

This issue is still happening.
Pulumi have saved token in resources and use token which was used in first deployment. On second deployment or if vault token expired you will received error 403. Pulumi do not get actual token, but force to download old token from stack.
Should get new token from EVN VAULT_TOKEN. This is a bug.