Errors with `aws:route53/getZone:getZone` after upgrading to 5.14.0 #2144

henriiik · 2022-09-19T11:31:37Z

What happened?

When i upgraded my project to 5.14.0 i started getting the following error a lot (20+ times) when i run a preview. Downgrading to 5.13.0 makes the problem go away.

pulumi:pulumi:Stack (stack-name):
  error: Running program '/path/to/project' failed with an unhandled exception:
  Error: invocation of aws:route53/getZone:getZone returned an error: unable to validate AWS credentials. Make sure you have:
  
      • Set your AWS region, e.g. `pulumi config set aws:region us-west-2`
      • Configured your AWS credentials as per https://pulumi.io/install/aws.html
      You can also set these via cli using `aws configure`.
  
  
      at Object.callback (/path/to/project/node_modules/@pulumi/runtime/invoke.ts:159:33)
      at Object.onReceiveStatus (/path/to/project/node_modules/@grpc/grpc-js/src/client.ts:338:26)
      at Object.onReceiveStatus (/path/to/project/node_modules/@grpc/grpc-js/src/client-interceptors.ts:426:34)
      at Object.onReceiveStatus (/path/to/project/node_modules/@grpc/grpc-js/src/client-interceptors.ts:389:48)
      at /path/to/project/node_modules/@grpc/grpc-js/src/call-stream.ts:276:24
      at processTicksAndRejections (node:internal/process/task_queues:77:11)

Steps to reproduce

upgrade from 5.13.0 to 5.14.0
run pulumi up

Expected Behavior

I should get a preview of the my changes

Actual Behavior

The process crashes.

Output of `pulumi about`

No response

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

The text was updated successfully, but these errors were encountered:

stack72 · 2022-09-19T11:52:40Z

Hi @henriiik

How are you authenticating against your provider?

Paul

cyberox · 2022-09-19T11:55:35Z

Hi @stack72,
I'm having the same issue. All resources that depend on a custom provider are failing when using version 5.14.0, with version 5.13.0 this works correctly:

    const csProvider = new aws.Provider(
      "central-services",
      {
        assumeRole: {
          roleArn: `arn:aws:iam::${args.accountId}:role/OrganizationAccountAccessRole`,
        },
        region: config.region,
      }
    );

    const vpcIpam = new aws.ec2.VpcIpam(
      `${name}`,
      {
        operatingRegions: [
          {
            regionName: config.region,
          },
        ],
        description: "Default",
      },
      {
        provider: csProvider,
      }
    );

stack72 · 2022-09-19T11:58:07Z

As a way to unblock your deployments, please set skipCredentialsValidation to be true in either the custom provider or via config while we investigate this

henriiik · 2022-09-19T12:10:04Z

@stack72 thanks for the quick reply!

We use environment variables (generated by aws-vault) for the credentials and then we pass in a role ARN to the provider.

setting skipCredentialsValidation: true worked as a workaround. Thanks!

stack72 · 2022-09-19T12:17:28Z

Are you able to tell me what environment variables are being set? (Just the names)

henriiik · 2022-09-19T12:19:00Z

These are the ones i have set

AWS_PAGER
AWS_VAULT
AWS_DEFAULT_REGION
AWS_REGION
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN
AWS_SECURITY_TOKEN
AWS_SESSION_EXPIRATION

guineveresaenger · 2022-09-19T22:15:28Z

heya @henriiik -

I'm struggling to repro this locally using a pre-setup Route 53 and the following bit of code:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";


const getZone = aws.route53.getZone({
    name: "test.com-9dc4e00",
});

export let zone = getZone

My default AWS profile sets the region:

@guin:.aws🦉 cat config
[default]
region = us-west-2

My env vars are:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN

guineveresaenger · 2022-09-19T22:32:31Z

Note that if I remove my default credentials, I do get the error you see. But as soon as I set the env var, it works again:

henriiik · 2022-09-20T03:41:06Z

Hello!

You need to have the provider assume a role like in the example @cyberox posted above.

I was able to create a minimal repo and push it to this repo: https://github.com/henriiik/aws-provider-bug

first preview works, since role is not created yet. However the update fails.

$ pulumi up
Previewing update (dev)

View Live: https://app.pulumi.com/henriiik/aws-provider-bug/dev/previews/8c188fb4-6197-411b-a6df-6faeee1e6c01

     Type                     Name                  Plan       Info
 +   pulumi:pulumi:Stack      aws-provider-bug-dev  create     6 messages
 +   ├─ aws:iam:Role          role                  create     
 +   └─ pulumi:providers:aws  provider              create     

Do you want to perform this update? yes
Updating (dev)

View Live: https://app.pulumi.com/henriiik/aws-provider-bug/dev/updates/1

     Type                     Name                  Status                  Info
 +   pulumi:pulumi:Stack      aws-provider-bug-dev  **creating failed**     1 error; 6 messages
 +   ├─ aws:iam:Role          role                  created                 
 +   └─ pulumi:providers:aws  provider              created                 
 
Diagnostics:
  pulumi:pulumi:Stack (aws-provider-bug-dev):
    error: Running program '/Users/henke/dev/aws-provider-bug' failed with an unhandled exception:
    Error: invocation of aws:route53/getZone:getZone returned an error: unable to validate AWS credentials. Make sure you have:
    
     	 • Set your AWS region, e.g. `pulumi config set aws:region us-west-2`
     	 • Configured your AWS credentials as per https://pulumi.io/install/aws.html
     	 You can also set these via cli using `aws configure`.
    
    
        at Object.callback (/Users/henke/dev/aws-provider-bug/node_modules/@pulumi/runtime/invoke.ts:159:33)
        at Object.onReceiveStatus (/Users/henke/dev/aws-provider-bug/node_modules/@grpc/grpc-js/src/client.ts:338:26)
        at Object.onReceiveStatus (/Users/henke/dev/aws-provider-bug/node_modules/@grpc/grpc-js/src/client-interceptors.ts:426:34)
        at Object.onReceiveStatus (/Users/henke/dev/aws-provider-bug/node_modules/@grpc/grpc-js/src/client-interceptors.ts:389:48)
        at /Users/henke/dev/aws-provider-bug/node_modules/@grpc/grpc-js/src/call-stream.ts:276:24
        at processTicksAndRejections (node:internal/process/task_queues:78:11)
 
Resources:
    + 3 created

Duration: 6s

subsequent preview fail

$ pulumi up
Previewing update (dev)

View Live: https://app.pulumi.com/henriiik/aws-provider-bug/dev/previews/6275c9b0-5938-4d0a-8429-b69a866d8791

     Type                 Name                  Plan     Info
     pulumi:pulumi:Stack  aws-provider-bug-dev           1 error
 
Diagnostics:
  pulumi:pulumi:Stack (aws-provider-bug-dev):
    error: Running program '/Users/henke/dev/aws-provider-bug' failed with an unhandled exception:
    Error: invocation of aws:route53/getZone:getZone returned an error: unable to validate AWS credentials. Make sure you have:
    
     	 • Set your AWS region, e.g. `pulumi config set aws:region us-west-2`
     	 • Configured your AWS credentials as per https://pulumi.io/install/aws.html
     	 You can also set these via cli using `aws configure`.
    
    
        at Object.callback (/Users/henke/dev/aws-provider-bug/node_modules/@pulumi/runtime/invoke.ts:159:33)
        at Object.onReceiveStatus (/Users/henke/dev/aws-provider-bug/node_modules/@grpc/grpc-js/src/client.ts:338:26)
        at Object.onReceiveStatus (/Users/henke/dev/aws-provider-bug/node_modules/@grpc/grpc-js/src/client-interceptors.ts:426:34)
        at Object.onReceiveStatus (/Users/henke/dev/aws-provider-bug/node_modules/@grpc/grpc-js/src/client-interceptors.ts:389:48)
        at /Users/henke/dev/aws-provider-bug/node_modules/@grpc/grpc-js/src/call-stream.ts:276:24
        at processTicksAndRejections (node:internal/process/task_queues:78:11)

richmeij · 2022-09-20T08:42:14Z

Just wanted to weigh in by confirming that setting skipCredentialsValidation to true still gives the following error when using assumeRole in a custom Provider during preview:

Previewing update....
    ~ pulumi:providers:aws: (update)
        [id=xxx]
        [urn=urn:pulumi:Drone-prod::Drone::pulumi:providers:aws::aws]
        assumeRole               : (json) {
            roleArn: "arn:aws:iam::123456789:role/PulumiDeployRole"
        }

        region                   : "eu-central-1"
      ~ skipCredentialsValidation: "false" => "true"
        skipGetEc2Platforms      : "true"
        skipMetadataApiCheck     : "true"
        skipRegionValidation     : "true"
      ~ version                  : "5.13.0" => "5.14.0"
error: unable to validate AWS credentials. Make sure you have: 

 	 • Set your AWS region, e.g. `pulumi config set aws:region us-west-2` 
 	 • Configured your AWS credentials as per https://pulumi.io/install/aws.html 
 	 You can also set these via cli using `aws configure`.

edit: We use the following environment vars:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_REGION
PULUMI_ACCESS_TOKEN

We pass optional settings to the provider. In this case, only the assumeRole and region settings are available and passed to the Provider constructor. The AWS account which corresponds to the accesskey/secretkey set in the env has all the rights needed, and can assume the role specified (this all works prior ro 5.14.0)

  const regionToUse = region || awsConfig.get('region') || config.require('region');
  const roleToUse = awsConfig.getObject('assumeRole');
  const awsAccessKey = config.getSecret('awsAccessKey');
  const awsSecretKey = config.getSecret('awsSecretKey');

  const provider = new aws.Provider('aws', {
    region: <aws.Region>regionToUse,
    accessKey: awsAccessKey,
    secretKey: awsSecretKey,
    assumeRole: <aws.types.input.ProviderAssumeRole>roleToUse,
    skipCredentialsValidation: true,
  });

guineveresaenger · 2022-09-20T20:06:16Z

thank you for the additional background @henriiik - this seems to be specific to custom Providers. I'm running a few scenarios to determine why credentials are not being propagated correctly.

guineveresaenger · 2022-09-20T23:16:56Z

All right, so this is my current presumption of what happened.

In pulumi-aws v5.14, we pulled in changes to the upstream config verification library, which added a verification check for the roleArn property of assumeRole to address a silent ignore behavior.

With our upgrade to v5.14, we suddenly started hitting the error returned on this line, because we had not been propagating the roleArn value correctly.

The upside here is that the custom Provider's role assumption will be much more safe and reliable! Please hold tight while we work to properly set that config value and reconsider our error surfacing strategy for this new check.

henriiik · 2022-09-21T05:01:25Z

@guineveresaenger thanks for the update!

henriiik added kind/bug Some behavior is incorrect or out of spec needs-triage Needs attention from the triage team labels Sep 19, 2022

stack72 added awaiting-feedback Blocked on input from the author and removed needs-triage Needs attention from the triage team labels Sep 19, 2022

guineveresaenger self-assigned this Sep 19, 2022

guineveresaenger removed the awaiting-feedback Blocked on input from the author label Sep 19, 2022

guineveresaenger added this to the 0.78 milestone Sep 19, 2022

guineveresaenger added the needs-repro Needs repro steps before it can be triaged or fixed label Sep 19, 2022

guineveresaenger removed the needs-repro Needs repro steps before it can be triaged or fixed label Sep 20, 2022

guineveresaenger mentioned this issue Sep 21, 2022

Read in role ARN in preConfigure validation #2149

Merged

pasqualet mentioned this issue Oct 4, 2022

Can't create Kubernetes cluster cloud-toolkit/cloud-toolkit-aws#16

Closed

viveklak closed this as completed in #2149 Oct 4, 2022

pulumi-bot added the resolution/fixed This issue was fixed label Oct 4, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Errors with `aws:route53/getZone:getZone` after upgrading to 5.14.0 #2144

Errors with `aws:route53/getZone:getZone` after upgrading to 5.14.0 #2144

henriiik commented Sep 19, 2022

stack72 commented Sep 19, 2022

cyberox commented Sep 19, 2022 •

edited by stack72

Loading

stack72 commented Sep 19, 2022

henriiik commented Sep 19, 2022

stack72 commented Sep 19, 2022

henriiik commented Sep 19, 2022

guineveresaenger commented Sep 19, 2022 •

edited

Loading

guineveresaenger commented Sep 19, 2022 •

edited

Loading

henriiik commented Sep 20, 2022

richmeij commented Sep 20, 2022 •

edited

Loading

guineveresaenger commented Sep 20, 2022

guineveresaenger commented Sep 20, 2022

henriiik commented Sep 21, 2022

Errors with aws:route53/getZone:getZone after upgrading to 5.14.0 #2144

Errors with aws:route53/getZone:getZone after upgrading to 5.14.0 #2144

Comments

henriiik commented Sep 19, 2022

What happened?

Steps to reproduce

Expected Behavior

Actual Behavior

Output of pulumi about

Additional context

Contributing

stack72 commented Sep 19, 2022

cyberox commented Sep 19, 2022 • edited by stack72 Loading

stack72 commented Sep 19, 2022

henriiik commented Sep 19, 2022

stack72 commented Sep 19, 2022

henriiik commented Sep 19, 2022

guineveresaenger commented Sep 19, 2022 • edited Loading

guineveresaenger commented Sep 19, 2022 • edited Loading

henriiik commented Sep 20, 2022

richmeij commented Sep 20, 2022 • edited Loading

guineveresaenger commented Sep 20, 2022

guineveresaenger commented Sep 20, 2022

henriiik commented Sep 21, 2022

Errors with `aws:route53/getZone:getZone` after upgrading to 5.14.0 #2144

Errors with `aws:route53/getZone:getZone` after upgrading to 5.14.0 #2144

Output of `pulumi about`

cyberox commented Sep 19, 2022 •

edited by stack72

Loading

guineveresaenger commented Sep 19, 2022 •

edited

Loading

guineveresaenger commented Sep 19, 2022 •

edited

Loading

richmeij commented Sep 20, 2022 •

edited

Loading