(pipelines): "cannot consume a cross reference from stack" when using CDK Pipelines

[nasuboots]

Deploying a subclass of Stage with CdkPipeline to a different region fails because of cdk synth error.

Reproduction Steps

#!/usr/bin/env node
import 'source-map-support/register';

import { App, Construct, Stage, Stack, StackProps, StageProps, SecretValue } from '@aws-cdk/core';
import { CdkPipeline, SimpleSynthAction } from '@aws-cdk/pipelines';
import * as codepipeline from '@aws-cdk/aws-codepipeline';
import * as codepipeline_actions from '@aws-cdk/aws-codepipeline-actions';
import * as dynamodb from '@aws-cdk/aws-dynamodb';

class DatabaseStack extends Stack {
  constructor(scope: Construct, id: string, props?: StageProps) {
    super(scope, id, props);

    new dynamodb.Table(this, 'Table', {
      partitionKey: { name: 'PK', type: dynamodb.AttributeType.STRING },
    });
  }
}

class MyApplication extends Stage {
  constructor(scope: Construct, id: string, props?: StageProps) {
    super(scope, id, props);

    new DatabaseStack(this, 'Database');
  }
}

class AwsCdkPipelinesCrossRegionStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const sourceArtifact = new codepipeline.Artifact();
    const cloudAssemblyArtifact = new codepipeline.Artifact();

    const pipeline = new CdkPipeline(this, 'Pipeline', {
      pipelineName: 'MyAppPipeline',
      cloudAssemblyArtifact,

      sourceAction: new codepipeline_actions.GitHubSourceAction({
        actionName: 'GitHub',
        output: sourceArtifact,
        oauthToken: SecretValue.secretsManager('GITHUB_TOKEN_NAME'),
        owner: 'OWNER',
        repo: 'REPO',
      }),

      synthAction: SimpleSynthAction.standardNpmSynth({
        sourceArtifact,
        cloudAssemblyArtifact,
      }),
    });

    pipeline.addApplicationStage(new MyApplication(this, 'Prod', {
      // stage env
      env: {
        account: '111111111111',
        region: 'eu-west-1',
      },
    }));
  }
}

const app = new App();
new AwsCdkPipelinesCrossRegionStack(app, 'AwsCdkPipelinesCrossRegionStack', {
  // pipeline env
  env: {
    account: '111111111111',
    region: 'us-east-1',
  },
});

$ npm run cdk synth

> aws-cdk-pipelines-cross-region@0.1.0 cdk /workspaces/aws-cdk-pipelines-cross-region
> cdk "synth"


/workspaces/aws-cdk-pipelines-cross-region/node_modules/@aws-cdk/core/lib/private/refs.ts:57
    throw new Error(
          ^
Error: Stack "AwsCdkPipelinesCrossRegionStack" cannot consume a cross reference from stack "cross-region-stack-111111111111:eu-west-1". Cross stack references are only supported for stacks deployed to the same environment or between nested stacks and their parent stack
    at resolveValue (/workspaces/aws-cdk-pipelines-cross-region/node_modules/@aws-cdk/core/lib/private/refs.ts:57:11)
    at Object.resolveReferences (/workspaces/aws-cdk-pipelines-cross-region/node_modules/@aws-cdk/core/lib/private/refs.ts:33:24)
    at Object.prepareApp (/workspaces/aws-cdk-pipelines-cross-region/node_modules/@aws-cdk/core/lib/private/prepare-app.ts:31:3)
    at Object.synthesize (/workspaces/aws-cdk-pipelines-cross-region/node_modules/@aws-cdk/core/lib/private/synthesis.ts:24:3)
    at App.synth (/workspaces/aws-cdk-pipelines-cross-region/node_modules/@aws-cdk/core/lib/stage.ts:188:23)
    at process.<anonymous> (/workspaces/aws-cdk-pipelines-cross-region/node_modules/@aws-cdk/core/lib/app.ts:123:45)
    at Object.onceWrapper (events.js:422:26)
    at process.emit (events.js:315:20)
    at process.EventEmitter.emit (domain.js:467:12)
    at process.emit (/workspaces/aws-cdk-pipelines-cross-region/node_modules/source-map-support/source-map-support.js:495:21)

What did you expect to happen?

Generating a template succeeded.

What actually happened?

pipeline env	stage env	result
111111111111, us-east-1	111111111111, us-east-1	Successfully synthesized to /workspaces/aws-cdk-pipelines-cross-region/cdk.out
111111111111, us-east-1	222222222222, us-east-1	Successfully synthesized to /workspaces/aws-cdk-pipelines-cross-region/cdk.out
111111111111, us-east-1	111111111111, eu-west-1	Error: Stack "AwsCdkPipelinesCrossRegionStack" cannot consume a cross reference from stack "cross-region-stack-111111111111:eu-west-1"
111111111111, us-east-1	222222222222, eu-west-1	Error: Stack "AwsCdkPipelinesCrossRegionStack" cannot consume a cross reference from stack "cross-region-stack-111111111111:eu-west-1"

Environment

• CDK CLI Version : v1.87.0
• Framework Version: v1.87.0
• Node.js Version: v14.15.4
• OS : Debian GNU/Linux 10 (buster)
• Language (Version): TypeScript (3.9.7)

Other

cdk version	result
v1.76	Successfully synthesized to /workspaces/aws-cdk-pipelines-cross-region/cdk.out
v1.77	Successfully synthesized to /workspaces/aws-cdk-pipelines-cross-region/cdk.out
v1.78 ~	Error: Stack "AwsCdkPipelinesCrossRegionStack" cannot consume a cross reference from stack "cross-region-stack-111111111111:eu-west-1"

---

This is 🐛 Bug Report

[markusbecker]

Uff, i though me was the problem. Having the same issue. If stage region != pipeline region i get the same error. (1.87.1)

[rix0rrr]

Unfortunately does not reproduce for me. Using the code above, and using the following versions:

1.78.0 (version mentioned in the title)
1.87.0 (version mentioned in the "environment" block)
1.83.0 (just one I happened to have lying around)

And then:

$ npx cdk synth
Successfully synthesized to /Users/huijbers/Temp/testpipeline/cdk.out
Supply a stack id (cross-region-stack-111111111111:eu-west-1, AwsCdkPipelinesCrossRegionStack) to display its template.

[rix0rrr]

Have you cut something out of your code sample that is causing the actual issue?

[rix0rrr]

Is the issue perhaps that you have ^ dependencies somewhere, and/or there are multiple copies of the CDK core library in your dependency tree?

grep '\^' package.json     # <-- should NOT show any CDK libraries
find . -path \*/@aws-cdk/core   # <-- should only find ONE directory

[tompiscitell]

I'm running into the same problem and I can confirm I don't have multiple copies of the CDK core library in my dependency tree.

grep '\^' package.json 
    "@types/jest": "^26.0.10",
    "jest": "^26.4.2",
    "ts-jest": "^26.2.0",
    "ts-node": "^9.0.0",
    "source-map-support": "^0.5.16"

find . -path \*/@aws-cdk/core  
./node_modules/@aws-cdk/core

I pushed the code I used to reproduce this here: https://github.com/tompiscitell/test-cdk-pipeline. It has branches with versions 1.77.0 and 1.88.0 in addition to 1.87.1 (master). I'm seeing the same behavior as @nasuboots in that it works on 1.77.0 and doesn't work on any of the later version.

[mathewlk]

+1 - I've tried versions 1.86 and 1.88 and am seeing the same issue.

[oleksii-donoha]

I have the same problem with Python libs version 1.88. Both environments are bootstrapped.
Regarding stale libs, I don't think I have any since all CDK libs are frozen to 1.88 in setup.py and this issue can be reproduced both on local machine and clean CodeBuild environment.

[rix0rrr]

Ha haaaah! Found the culprit!

    "@aws-cdk/aws-kms:defaultKeyPolicies": true,

Removing that makes it work.

Thanks for the repro @tompiscitell, that was extremely helpful. Still not quite sure why the error didn't repro for me, but now we have a way to investigate.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.