Cannot determine scope for context provider availability-zones for China regions

[bnusunny]

Current release of CDK (0.18.1) Cannot determine scope for context provider availability-zones for China regions (cn-north-1 and cn-northwest-1).

I did setup default credentials with aws configure and can list S3 buckets without problem. But CDK was not able to get the account and region.

sunhua@haorld_mac ~/Documents/Projects/github.com/nwcdlabs/openshift-on-aws-cn/scripts/infrastructure                [10:13:39]
> $ aws configure                                                                                              [±origin-3.11 ●]
AWS Access Key ID [****************TEMA]:
AWS Secret Access Key [****************RXxI]:
Default region name [cn-northwest-1]:
Default output format [json]:

sunhua@haorld_mac ~/Documents/Projects/github.com/nwcdlabs/openshift-on-aws-cn/scripts/infrastructure                [10:13:47]
> $ aws s3 ls                                                                                                  [±origin-3.11 ●]
2018-07-23 11:23:08 aws-logs-215043275130-cn-northwest-1
2018-08-14 13:44:13 cf-templates-m067e3zuzu6h-cn-north-1
2018-06-11 16:34:18 cf-templates-m067e3zuzu6h-cn-northwest-1
2018-11-11 19:13:29 cloudtrail-zhy-201811111912
2018-07-06 10:40:56 config-bucket-215043275130
2018-11-11 19:05:59 config-bucket-bjs-215043275130


sunhua@haorld_mac ~/Documents/Projects/github.com/nwcdlabs/openshift-on-aws-cn/scripts/infrastructure                [10:13:51]
> $ cdk diff                                                                                                   [±origin-3.11 ●]
[Error at /InfrastructureStack/okd_vpc] Cannot determine scope for context provider availability-zones with props: account=undefined,region=undefined.
This usually happens when AWS credentials are not available and the default account/region cannot be determined.
[Error at /InfrastructureStack/okd_vpc] Cannot determine scope for context provider availability-zones with props: account=undefined,region=undefined.
This usually happens when AWS credentials are not available and the default account/region cannot be determined.
Found errors

sunhua@haorld_mac ~/Documents/Projects/github.com/nwcdlabs/openshift-on-aws-cn/scripts/infrastructure                [10:14:08]
> $ cdk --version                                                                                              [±origin-3.11 ●]
0.18.1 (build 9f7af21)

[rix0rrr]

Please post a trace of executing with -v enabled.

[bnusunny]

Here is the debug log. I think there might be an issue with STS endpoint for China region.

> $ cdk diff -v                                                                                                                                                [±origin-3.11 ●]
CDK toolkit version: 0.18.1 (build 9f7af21)
Command line arguments: { _: [ 'diff' ],
  trace: false,
  strict: false,
  'ignore-errors': false,
  ignoreErrors: false,
  json: false,
  j: false,
  verbose: true,
  v: true,
  ec2creds: undefined,
  i: undefined,
  'version-reporting': undefined,
  versionReporting: undefined,
  'path-metadata': true,
  pathMetadata: true,
  version: false,
  help: false,
  'role-arn': undefined,
  r: undefined,
  roleArn: undefined,
  '$0': '/Users/sunhua/.config/yarn/global/node_modules/.bin/cdk',
  app: undefined,
  context: undefined,
  plugin: undefined,
  rename: undefined,
  profile: undefined,
  proxy: undefined,
  template: undefined }
Determining whether we're on an EC2 instance.
Does not look like EC2 instance.
cdk.json: {
  "app": "node bin/infrastructure.js",
  "versionReporting": false
}
Setting "aws:cdk:toolkit:default-region" context to cn-northwest-1
Resolving default credentials
Looking up default account ID from STS
Unable to determine the default AWS account (did you configure "aws configure"?): { InvalidClientTokenId: The security token included in the request is invalid.
    at Request.extractError (/Users/sunhua/.config/yarn/global/node_modules/aws-sdk/lib/protocol/query.js:47:29)
    at Request.callListeners (/Users/sunhua/.config/yarn/global/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/Users/sunhua/.config/yarn/global/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/Users/sunhua/.config/yarn/global/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/Users/sunhua/.config/yarn/global/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/Users/sunhua/.config/yarn/global/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /Users/sunhua/.config/yarn/global/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/Users/sunhua/.config/yarn/global/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/Users/sunhua/.config/yarn/global/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/Users/sunhua/.config/yarn/global/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
  message: 'The security token included in the request is invalid.',
  code: 'InvalidClientTokenId',
  time: 2018-11-30T17:03:31.900Z,
  requestId: '03dcba49-f4c2-11e8-8182-099c531174c2',
  statusCode: 403,
  retryable: false,
  retryDelay: 2.5160029693942487 }
Setting "aws:cdk:toolkit:default-account" context to undefined
context: { 'aws:cdk:toolkit:default-region': 'cn-northwest-1',
  'aws:cdk:toolkit:default-account': undefined,
  'aws:cdk:enable-path-metadata': true }
outdir: /var/folders/2t/fwc412dx2kd671g9dcyvbw5mb9vjhh/T/cdkFIVLoS
asg AutoScalingGroup [InfrastructureStack/onebox-asg] is created.
outfile: /var/folders/2t/fwc412dx2kd671g9dcyvbw5mb9vjhh/T/cdkFIVLoS/cdk.out
{ version: '0.14.0',
  stacks:
   [ { name: 'InfrastructureStack',
       environment: [Object],
       template: [Object],
       metadata: [Object] } ],
  runtime:
   { libraries:
      { infrastructure: '0.1.0',
        '@aws-cdk/aws-ec2': '0.18.1',
        '@aws-cdk/cdk': '0.18.1',
        '@aws-cdk/cx-api': '0.18.1',
        '@aws-cdk/aws-autoscaling': '0.18.1',
        '@aws-cdk/aws-elasticloadbalancingv2': '0.18.1',
        '@aws-cdk/aws-cloudwatch': '0.18.1',
        '@aws-cdk/aws-iam': '0.18.1',
        '@aws-cdk/aws-codedeploy-api': '0.18.1',
        '@aws-cdk/aws-autoscaling-common': '0.18.1' } } }
Removing outdir /var/folders/2t/fwc412dx2kd671g9dcyvbw5mb9vjhh/T/cdkFIVLoS
[Error at /InfrastructureStack/okd_vpc] Cannot determine scope for context provider availability-zones with props: account=undefined,region=cn-northwest-1.
This usually happens when AWS credentials are not available and the default account/region cannot be determined.
  VpcNetwork.addError (/Users/sunhua/Documents/Projects/github.com/nwcdlabs/openshift-on-aws-cn/scripts/infrastructure/node_modules/@aws-cdk/cdk/lib/core/construct.js:202:21)
  ContextProvider.getStringListValue (/Users/sunhua/Documents/Projects/github.com/nwcdlabs/openshift-on-aws-cn/scripts/infrastructure/node_modules/@aws-cdk/cdk/lib/context.js:78:26)
  AvailabilityZoneProvider.get availabilityZones [as availabilityZones] (/Users/sunhua/Documents/Projects/github.com/nwcdlabs/openshift-on-aws-cn/scripts/infrastructure/node_modules/@aws-cdk/cdk/lib/context.js:116:30)
  new VpcNetwork (/Users/sunhua/Documents/Projects/github.com/nwcdlabs/openshift-on-aws-cn/scripts/infrastructure/node_modules/@aws-cdk/aws-ec2/lib/vpc.js:94:73)
  new OneBoxStack (/Users/sunhua/Documents/Projects/github.com/nwcdlabs/openshift-on-aws-cn/scripts/infrastructure/bin/infrastructure.js:11:21)
  Object.<anonymous> (/Users/sunhua/Documents/Projects/github.com/nwcdlabs/openshift-on-aws-cn/scripts/infrastructure/bin/infrastructure.js:51:1)
  Module._compile (module.js:652:30)
  Object.Module._extensions..js (module.js:663:10)
  Module.load (module.js:565:32)
  tryModuleLoad (module.js:505:12)
  Function.Module._load (module.js:497:3)
  Function.Module.runMain (module.js:693:10)
  startup (bootstrap_node.js:188:16)
  bootstrap_node.js:609:3
[Error at /InfrastructureStack/okd_vpc] Cannot determine scope for context provider availability-zones with props: account=undefined,region=cn-northwest-1.
This usually happens when AWS credentials are not available and the default account/region cannot be determined.
  VpcNetwork.addError (/Users/sunhua/Documents/Projects/github.com/nwcdlabs/openshift-on-aws-cn/scripts/infrastructure/node_modules/@aws-cdk/cdk/lib/core/construct.js:202:21)
  ContextProvider.getStringListValue (/Users/sunhua/Documents/Projects/github.com/nwcdlabs/openshift-on-aws-cn/scripts/infrastructure/node_modules/@aws-cdk/cdk/lib/context.js:78:26)
  AvailabilityZoneProvider.get availabilityZones [as availabilityZones] (/Users/sunhua/Documents/Projects/github.com/nwcdlabs/openshift-on-aws-cn/scripts/infrastructure/node_modules/@aws-cdk/cdk/lib/context.js:116:30)
  VpcNetwork.createSubnets (/Users/sunhua/Documents/Projects/github.com/nwcdlabs/openshift-on-aws-cn/scripts/infrastructure/node_modules/@aws-cdk/aws-ec2/lib/vpc.js:165:62)
  new VpcNetwork (/Users/sunhua/Documents/Projects/github.com/nwcdlabs/openshift-on-aws-cn/scripts/infrastructure/node_modules/@aws-cdk/aws-ec2/lib/vpc.js:103:14)
  new OneBoxStack (/Users/sunhua/Documents/Projects/github.com/nwcdlabs/openshift-on-aws-cn/scripts/infrastructure/bin/infrastructure.js:11:21)
  Object.<anonymous> (/Users/sunhua/Documents/Projects/github.com/nwcdlabs/openshift-on-aws-cn/scripts/infrastructure/bin/infrastructure.js:51:1)
  Module._compile (module.js:652:30)
  Object.Module._extensions..js (module.js:663:10)
  Module.load (module.js:565:32)
  tryModuleLoad (module.js:505:12)
  Function.Module._load (module.js:497:3)
  Function.Module.runMain (module.js:693:10)
  startup (bootstrap_node.js:188:16)
  bootstrap_node.js:609:3
Found errors
Error: Found errors
    at AppStacks.synthesizeStacks (/Users/sunhua/.config/yarn/global/node_modules/aws-cdk/lib/api/cxapp/stacks.ts:111:15)
    at <anonymous>

[rix0rrr]

I cannot reproduce this on my machine using the steps your described. Could you have a look at the files in your ~/.aws directory, and confirm that they look like this:

~/.aws/credentials

[default]
aws_access_key_id = AKIAXXXXXX
aws_secret_access_key = XXXXXXX

~/.aws/config

[default]
output = json
region = cn-northwest-1

And that the shell in which you were running this did not have any AWS_ environment variables.

$ env | grep AWS_
(should not give any output)

[bnusunny]

I did check these things. It works on aws cli since I can list s3 buckets in China regions. I have additional profiles setup for Global regions. Will that cause the problem?

[bnusunny]

I see in the line here , region parameter is not set. The client will connect to STS in us-east-1, not China region, right?

const result = await new AWS.STS({ credentials: creds }).getCallerIdentity().promise();

[rix0rrr]

I know, I've noticed the same, and also thought it was a bug. Yet when I tried to reproduce failures, it did work on my machine. I don't know why.

Here's my bug report to SDK team: aws/aws-sdk-js#2377

[rix0rrr]

I also have multiple profiles in my AWS config. The China endpoint does work for me, if credentials configured either in the default or a specific profile.

[rix0rrr]

If it helps, for me I'm at this version of the SDK:

$ cat node_modules/aws-sdk/package.json  | grep version
  "version": "2.356.0"

[rix0rrr]

Okay, I now know this happens to work on my machine because somewhere in my Node process the following gets set:

AWS_SDK_LOAD_CONFIG = 1

Which makes the default region load itself from the .ini file, and makes the rest work out. Apparently this is not true for everyone.

[rix0rrr]

Ah and I have that variable set because I'm loading a credentials plugin!!

[rix0rrr]

Your proposed fix is correct. We need to thread the region argument through to the STS client in sdk.ts.

This won't make it work for AssumeRole profiles yet. To that end, we need to configure the profile region globally in AWS.config as soon as we have it, and tell the JS SDK team about the assume role bug.