route53.HostedZone.fromLookup returned error

[gsm1011]

I have a hosted zone on route53, and use the following CDK code (typescript) to look up that zone using domainName, but got error while running cdk synth, cdk diff and cdk deploy. After changing to use route53.HostedZone.fromHostedZoneAttributes(), it works.

Reproduction Steps

const zone = route53.HostedZone.fromLookup(this, 'Zone', { domainName: props.domainName });

Error Log

Cannot retrieve value from context provider hosted-zone since account/region are not specified at the stack level. Either configure "env" with explicit account and region when you define your stack, or use the environment variables "CDK_DEFAULT_ACCOUNT" and "CDK_DEFAULT_REGION" to inherit environment information from the CLI (not recommended for production stacks)
Subprocess exited with error 1

Environment

• CLI Version : 1.19.0 (build 5597bbe)
• Framework Version:1.19.0
• OS :macOS Mojave version 10.14.6
• **Language :typescript **

Other

---

This is 🐛 Bug Report

[moofish32]

Hi @gsm1011

The error is pointing you in the right direction. When using:

const zone = route53.HostedZone.fromLookup(this, 'Zone', { domainName: props.domainName });

The CDK uses a context provider. The providers actually use the AWS SDK to call the Route53 API in this situation. In order for the SDK calls to work there must be credentials. Due to the potential errors that can occur with default credentials in your ENV the CDK requires that you set these variables either on the stack:

new MyStack(app, 'MyStackName', {
    env: {
        region: process.env.AWS_DEFAULT_REGION,
        account: '123456789012',
    },
});

or by setting the ENV variables described in the error message.

If you do one of those two things the fromLookup method will work.

[solshark]

That's weird but I'm facing same issue even env is defined during stack creation.

[gsm1011]

I had the same issue even after setting the env.

On Sun, Dec 29, 2019 at 12:00 PM solshark ***@***.***> wrote: That's weird but I'm facing same issue even env is defined during stack creation. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5547?email_source=notifications&email_token=AACSEFEXJEYK5N37JVLSNWDQ3D6XHA5CNFSM4J7A2IYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHZHEXA#issuecomment-569537116>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACSEFDH6IT5IWY2PPGD6PLQ3D6XHANCNFSM4J7A2IYA> .

-- Thanks, Shumin

[moofish32]

@gsm1011 @solshark -- can you guys provide a little more detail in the reproduction? The example I showed you is coming directly from a working source code that is using this method. (cdk version is 1.19.0). So I'm hoping we are just missing something easy for both of you. It's worth noting I am not using the CDK env vars, we are setting in the CDK App directly.

[shivlaks]

@gsm1011 @solshark - as @moofish32 mentioned, I'd also like to get a clearer idea of the repro steps to narrow this issue down. Does using env in your source work? Here's an example in the docs if you have not done it before

[codyfyi]

I just ran into this.

I had been relying on my ~/.aws/config and ~/.aws/credentials to deploy my stack to my default account and region. This has been working for my entire stack until I added route53.HostedZone.fromLookup.

I fixed the issue by setting the environment variables "CDK_DEFAULT_ACCOUNT" and "CDK_DEFAULT_REGION" as the error message suggested.

@shivlaks - Is it expected behavior that ~/.aws/config and ~/.aws/credentials are not sufficient for this operation?

[solshark]

@shivlaks sorry for late reply. Yep, I've tried to set env with no luck. Time was flying for that specific project so I went with:

    const hostedZone = PublicHostedZone.fromHostedZoneAttributes(
      this,
      'XXXXXX',
      { hostedZoneId: 'XXXXXX', zoneName: 'XXXXXX' }
    )

[optinirr]

The frustrating thing for me is that it works and deploys well but I can't get it to run the tests properly because of this issue.

[diogo-ap-old]

@gsm1011 @solshark - as @moofish32 mentioned, I'd also like to get a clearer idea of the repro steps to narrow this issue down. Does using env in your source work? Here's an example in the docs if you have not done it before

But why adding the region and account required only when using route53.HostedZone.fromLookup?

Cannot it be inferred like all the other commands?

[shivlaks]

@diogoap - fromLookup uses a context provider where a service call is made to Route53to retrieve the hostedZoneId given a domain name.

fromHostedZoneAttributes on the other hand requires that the hostedZoneId be provided and therefore does not need that lookup.

---

@optinirr can you help me with a repro of your issue (or trace if you can run cdk commands with the -v flag.

Here's what I'm doing (v1.41.0)
Stack 1:

new r53.HostedZone(this, 'my-hosted-zone', {
  zoneName: 'www.test.nala',
});

Stack 2 where I will look up the created hosted zone:

const imported = r53.HostedZone.fromLookup(this, 'my-looked-up-hz', {
  domainName: 'www.test.nala',
});

It's created by providing the environment. i.e.

new ReferenceRoute53Stack(app, 'ReferenceRoute53Stack', {
  env: {
    account: '123456789012',
    region: 'us-east-1',
  },
});

When I run cdk synth, the context is not there, it gets looked up and populated into cdk.context.json which looks like this

{
  "hosted-zone:account=123456789012:domainName=www.test.nala:region=us-east-1": {
    "Id": "/hostedzone/Z0577316FL60KM8IZ7KT",
    "Name": "www.test.nala."
  }
}

Additionally, the context lookup is not performed if the cdk.context.json is already populated. Context lookup is only done when the information is not available during synthesis.

[diogo-ap-old]

Thanks for the explanation @shivlaks
I understand that the account may be required for a call to retrieve the hostedZoneId
However, since the AWS credentials are already there, and it's related to some account, why not use the current account as default? This way we don't have to explicitly provide the account.

[shivlaks]

@diogoap There's a little more detail in our documentation on environments and why we require an environment to be established to use framework facilities such as context lookups.

to reduce friction during development you can do something like the following

new MyDevStack(app, 'dev', { 
  env: { 
    account: process.env.CDK_DEFAULT_ACCOUNT, 
    region: process.env.CDK_DEFAULT_REGION 
}});

These variables are set based on the AWS profile specified using the --profile option, or the default AWS profile if you don't specify one. This way you can switch your accounts or profiles and avoid having to specify an env on every iteration.

As mentioned in the documentation, this is useful during development, but probably an anti-pattern for production use.

[shivlaks]

adding the response requested label as this issue needs a repro.
Common gotchas have been configuration of env and the setup of credentials.

@optinirr - can you provide a minimal example where it works and is not usable in tests? Perhaps there's something actionable that we can follow up with.

[optinirr]

@shivlaks it was an env issue for me as well, as my env for deploy and test are not the same and my test env did not contain an "account" or "region".

[BryanPan342]

@shivlaks it was an env issue for me as well, as my env for deploy and test are not the same and my test env did not contain an "account" or "region".

@optinirr Can you help me wrap my head around the scope of the issue? My understanding is that fromLookup does a query to your AWS account for the HostedZoneId, which is why it is necessary to have environment variables declared as to provide context for the query.

If we look at the source code, we can see that fromLookup actually implements fromHostedZoneAttributes

aws-cdk/packages/@aws-cdk/aws-route53/lib/hosted-zone.ts

Lines 122 to 125 in 4e72d1e

	return HostedZone.fromHostedZoneAttributes(scope, id, {
	hostedZoneId: response.Id,
	zoneName: response.Name,
	});

If the hostedZoneId is known (which I assume it is because it is a test env), I think utilizing the fromHostedZoneAttributes method should suffice if adding an additional env is not desirable. Or potentially something similar to the documentation with different env variables could work as well.

new MyDevStack(app, 'dev', { 
  env: { 
    account: process.env.CDK_DEPLOY_ACCOUNT || process.env.CDK_DEFAULT_ACCOUNT, 
    region: process.env.CDK_DEPLOY_REGION || process.env.CDK_DEFAULT_REGION 
}});

Please lmk if I'm missing something though @shivlaks @optinirr!

[shivlaks]

@BryanPan342 I think the primary issue is confusion on when to use which API and the pre-requisites for each (local vs. requiring environment configuration). To reduce friction and avoid this common point of confusion, we need to shore up the README

Additionally, there's also fromHostedZoneId(...) and fromHostedZoneName(...) which will return an IHostedZone with just the ZoneId or the ZoneName field present. This is also done without having make an API call so it can be done entirely locally.

fromHostedZoneAttributes() returns an IHostedZone with both the hostedZoneId and the zoneName

finally, there's fromLookup() which populates context by making a service call to AWS through a context provider and stores it in cdk.context.json. If this is performed once and saved, subsequent calls will be cached and will not require making a service call.

If the entry does not exist in cdk.context.json, then it requires an environment to be configured and takes the attributes and performs an API call to determine which exact HostedZone is deployed and updates its behavior accordingly.

If hostedZoneNameServers property from an IHostedZone is desired, fromLookup() is the only fromXxX method that will support it.

[BryanPan342]

In this case, I think this is a documentation issue rather than a bug so changing the label to documentation and I will work on changing the documentation to make the use cases of each more clear.

[salsa2k]

I'm having some problems related to this.

When I try to use:

HostedZone.fromLookup(this, "HostedZone", {
      domainName: 'dev.mydomain.com',
    });

I got this error:

No hosted zone found with ID: Z0622398W2TQKSDOS30M (Service: AmazonRoute53; Status Code: 404; Error Code: NoSuchHostedZone; Request ID: 960c2b03-e42c-4c26-81ee-2f9254ca54dc

Not sure what is this Z0622398W2TQKSDOS30M since is not the same as my zone ID.

To verify if my domain and id's are ok, I have used:

aws route53 list-hosted-zones --profile poweruser 

Everything is in the list, and is also included in cdk.context.json

  "hosted-zone:account=0123456789012:domainName=dev.mydomain.com:region=us-east-1": {
    "Id": "/hostedzone/Z1APXXXXXXXXMJ",
    "Name": "dev.mydomain.com."
  }

[BryanPan342]

Not sure what is this Z0622398W2TQKSDOS30M since is not the same as my zone ID.

@salsa2k If you know the zone id, I suggest using the function fromHostedZoneAttributes, unless the hostedZoneNameServers is desired. I also would suggest to look whether the env variable you, hopefully, declared uses the account and region of profile poweruser.

For example, if you had a app configuration like this:

new MyDevStack(app, 'dev', { 
  env: { 
    account: process.env.CDK_DEPLOY_ACCOUNT || process.env.CDK_DEFAULT_ACCOUNT, 
    region: process.env.CDK_DEPLOY_REGION || process.env.CDK_DEFAULT_REGION 
}});

What I suspect might be happening is that CDK is looking at the wrong account and the query with that account is turning up empty responses.

To get around the above, you must export CDK_DEPLOY_ACCOUNT and CDK_DEPLOY_REGION manually (CDK wont autofill these from the aws config file).

From the documentation something like this should work

#!/bin/bash
# cdk-deploy-to.sh
export CDK_DEPLOY_ACCOUNT=$1
shift
export CDK_DEPLOY_REGION=$1
shift
cdk deploy "$@"

and then you can run bash cdk-deploy-to.sh 123457689 us-east-1!

Alternatively, I think you could also have your app configuration like this

new MyDevStack(app, 'dev', { 
  env: { 
    account: process.env.CDK_DEFAULT_ACCOUNT, 
    region: process.env.CDK_DEFAULT_REGION 
}});

and specify the profile during execution like this cdk deploy --profile poweruser

[salsa2k]

Hi, I found what is the problem:

Is related to my alias creatio.
Before I was using:

     new RecordSet(this, "ApiRecordSetA", {
       zone: props.zone,
       recordName: 'sub.domain.com',
       recordType: RecordType.A,
       target: RecordTarget.fromAlias(new ApiGatewayDomain(customDomain)),
     });

I changed it to:

    new ARecord(this, "ApiRecordA", {
      zone: props.zone,
      recordName: 'sub.domain.com',
      target: RecordTarget.fromAlias(new ApiGatewayDomain(customDomain)),
    });

And now its working o0

Thanks guys

[ilazakis]

cdk 2.29.0

I think the reason synth succeeds while deploy fails when using HostedZoneFromLookup is that the cdk optimistically assumes a successful lookup.

Here's the synth output for domainHostedZone.HostedZoneId() (using Golang):
DUMMY

That looks like a placeholder value in case of unsuccessful lookups.

In fact, after having fought this for a couple of hours, I think the answer to "why does the zone lookup sometimes fail" in my case is eventual consistency. After deleting and recreating the stack/zone in question a few times, I went for a walk, retried and the zone was now "found".

I have not looked into the JS or Golang implementations yet, but I suspect many may be falling victims to the CDK trying to fallback to a dummy zone instead of failing outright.