Cross-region/account references #49

rix0rrr · 2018-06-06T08:14:10Z

We were in the progress of defining how to transparently make this work.

Tracking it here because it's one of the most important things to pick up post-release.

rix0rrr · 2018-06-28T14:28:44Z

Use case: cross-region replication in combination with encryption.

Stack A -- defines the target bucket + a KMS key
Stack B -- defines the source bucket + an IAM role to use for replication.

For replication, the IAM role must must reference both buckets (local reference + one-way cross-stack reference)

To add encryption, the role must also reference the KMS Key, and the KMS key must also reference the role (bidirectional reference between stacks!)

rix0rrr · 2018-07-04T18:49:58Z

Note to self: if we build an ARN from a well-known name, we must not forget to manually add a dependency between the resources involved.

Probably, we should just link references between resources, which may then automatically translate into references between stacks.

skinny85 · 2018-07-06T00:02:48Z

Use case: cross-account CodePipeline.

Below is an example of a hacky way of defining a cross-account CodePipeline using CDK code today.
This follows the official blog post on the subject, but is a little simpler:
in the blog post, there were a total of 4 accounts, 2 used for deployment (test and prod).
I'm modelling a 2-stage Pipeline only, so I have 2 accounts: one with the CodeCommit repository in it,
and another where the CodeBuild project and the Pipeline itself live.

import { AccountPrincipal, App, ArnPrincipal, PolicyStatement, Stack } from 'aws-cdk';
import * as cp from 'aws-cdk-codepipeline';
import * as cb from 'aws-cdk-codebuild';
import * as cc from 'aws-cdk-codecommit';
import * as s3 from 'aws-cdk-s3';
import * as iam from 'aws-cdk-iam';
import * as kms from 'aws-cdk-kms';
import { codecommit } from 'aws-cdk-resources';
import { HackyIdentity } from "./hacky-identity";

const app = new App(process.argv);

const pipelineAcc = '123456789012';

/*************** the CodeCommit Repository Stack *************************/

const repoAcc = '012345678901';
const repoAccStack = new Stack(app, 'OtherAccStack', {
    env: {
        account: repoAcc,
    }
});

const repoName = 'CrossAccountNewRepo';
new cc.Repository(repoAccStack, 'OtherAccRepo', {
    repositoryName: repoName,
});

const repoRole = new iam.Role(repoAccStack, 'OtherAccRole', {
    roleName: 'CrossAccountRole',
    assumedBy: new AccountPrincipal(pipelineAcc),
});
repoRole.addToPolicy(new PolicyStatement()
    .addAllResources()
    .addActions('s3:*', 'codecommit:*', 'kms:*'));
const repoRoleArn = `arn:aws:iam::${repoAcc}:role/CrossAccountRole`;


/*************** the Pipeline Stack *************************/

const pipelineStack = new Stack(app, 'CrossAccountMainStack');

const key = new kms.EncryptionKey(pipelineStack, 'CrossAccountKmsKey');
const artifactBucket = new s3.Bucket(pipelineStack, 'CrossAccountPipelineBucket', {
    bucketName: 'codepipeline-us-west-2-cross-account-code-pipeline-bucket',
    encryptionKey: key,
    encryption: s3.BucketEncryption.Kms,
});
artifactBucket.grantReadWrite(
    new HackyIdentity(new ArnPrincipal(repoRoleArn)));

const pipeline = new cp.Pipeline(pipelineStack, 'CrossAccountPipeline', {
    pipelineName: 'CrossAccountPipeline',
    artifactBucket: artifactBucket,
});
pipeline.addToRolePolicy(new PolicyStatement()
    .addResource(repoRoleArn)
    .addActions('sts:AssumeRole'));

const sourceStage = new cp.Stage(pipeline, 'Source');
const sourceAction = new cp.CodeCommitSource(sourceStage, 'Source', {
    artifactName: 'SourceOutput',
    repository: cc.RepositoryRef.import(pipelineStack, 'ImportedRepo', {
        repositoryName: new codecommit.RepositoryName(repoName)
    }),
    roleArn: repoRoleArn,
});

const buildStage = new cp.Stage(pipeline, 'Build');
const codeBuildProject = new cb.BuildProject(pipelineStack, 'CrossAccountBuildProject', {
    source: new cb.CodePipelineSource(),
    artifacts: new cb.CodePipelineBuildArtifacts(),
    environment: {
        image: 'aws/codebuild/java:openjdk-8',
    },
    encryptionKey: key,
});
new cp.CodeBuildAction(buildStage, 'Build', {
    source: sourceAction,
    project: codeBuildProject,
});

process.stdout.write(app.run());

(this required adding the field roleArn to the CodePipeline Actions, and HackyIdentity is a very simple class:

import * as iam from 'aws-cdk-iam';
import { PolicyPrincipal, PolicyStatement } from 'aws-cdk';

class HackyIdentity implements iam.IIdentityResource {
    public readonly principal: PolicyPrincipal;

    constructor(principal: PolicyPrincipal) {
        this.principal = principal;
    }

    attachManagedPolicy(_: any): void {
    }

    attachInlinePolicy(_: iam.Policy): void {
    }

    addToPolicy(_: PolicyStatement): void {
    }
}

)

As can bee seen, I had to:

Give physical names to the IAM resources, like the Role in the CodeCommit Account that will be assumed by CodePipeline. This is in order to construct the ARNs needed to then be granted permissions
(that Role needs access to the Pipeline's S3 bucket, and the CodePipeline will assume it, so needs sts:AssumeRole for it).
I've actually given the CodeCommit Role * access to S3 - that's because I don't always know what's the ARN of the Bucket, as it can be managed by the L2 CodePipeline Construct (in this example, that's not the case, but in general, that's a problem - that means there's a circular dependency between resources in the 2 accounts). Same remark about the KMS key (that's actually required to make a cross-account CodePipeline work, otherwise the artifacts are encrypted with the default account key, and later stages in the Pipeline won't have access to the source retrieved from CodeCommit).

The dream experience would be something like:

// in the CodeCommit account / stack

const repository = new cc.Repository(repoAccStack, 'OtherAccRepo', {
    repositoryName: 'SomeName',
    // ...
});

// in the CodePipeline account / stack
new cp.CodeCommitSource(sourceStage, 'Source', {
    artifactName: 'SourceOutput',
    repository: repository,
});
// At this moment, the L2 CodePipeline construct notices that the Repository
// passed as the argument when constructing the `CodeCommitSource` is not from the same account as the Pipeline.
// In this case, it add the appropriate Role in the CodeCommit stack,
// adds the permissions for the CodePipeline Role to assume it,
// and makes sure that Role has access to the Pipeline's S3 Bucket,
// as well as the CodeCommit repository itself,
// and then sets it as the Role used in this Action.

I'm not sure how realistic it is to get this experience,
but I believe it's the best one we can provide for this case.

rix0rrr · 2018-07-06T08:37:25Z

Obligatory mention of #233, which defines a nice and general solution which also encompasses x-stack references.

rix0rrr · 2018-07-06T09:04:55Z

Yes, you're completely right. I have some cross-stack action going on right now and I'm having to do the same things. Just know that we're thinking about it & working on it! :)

As a general comment: in general we recommend defining a class for every Stack that you want to model. We don't always follow this recommendation ourselves, but only for small demoes and integ tests. Your code will automatically end up more organized with clearer interoperability points between the stacks if you divide them into classes.

eladb · 2018-07-12T12:03:25Z

@rix0rrr wrote:

Right now, if we're referencing objects between stacks, we can't use Ref and Fn::GetAtt anymore. Consumers need to be constantly cognisant of this, and drop down to physically naming their resources and constructing ARN strings to make these cross-references.

This breaks abstraction because you now need to know (when consuming a resource) whether it's a same-stack or a cross-stack resource.

I propose we build this concept into the framework (potentially at the L2 level): we'll introduce a new type of Token, called StackAwareToken, which will know what object it has been created for, and hence in what Stack that object lives. We'll also propagate the "current stack" while resolving Tokens.

Now, if the current stack != owning stack, the StackToken has the ability to

Throw a helpful error, because this will never work at CloudFormation deployment time.
Check whether the owning resource is physically named. If not, throw a helpful error. If it is, return the name or construct the ARN on the spot, whatever the case may be.

Doing this (as opposed to some Output/Export system, potentially involving Toolkit Context Providers) has the advantage that we can avoid ordering between stack deployments, which has two benefits:

Easier on the user; as long as you eventually deploy everything, the order doesn't matter.
We can do bidirectional references between stacks (which could never work if we needed to establish a total ordering between stack deploys), which is beneficial for things like IAM permissions.

There is the downside that potential changes in naming aren't automatically propagated/signaled to the user. For Exports, you would be PREVENTED from changing anything that a downstream stack depends on, such as a resource name. (For Outputs or SSM parameters there is no such constraint I think).

rix0rrr · 2018-07-12T12:06:54Z

The previous comment was written at a time where Stacks were front and center, but everything gets so much better if we automatically slice stacks.

When just build a more generic object graph (#233) and slice it down to stacks, we can do the work to transparently make this work at that point.

eladb · 2018-07-12T12:10:28Z

Agreed (just captured for posterity), however, even if we automatically slice stacks, there still needs to be a mechanism that resolves these cross-stack references, be them explicit or implicit

kennu · 2018-09-23T00:48:20Z

Use case: Route53 hostname records in a shared hosted zone

Stack A -- defines a CloudFront distribution in AWS account A (aaa.example.org)
Stack B -- defines a CloudFront distribution in AWS account B (bbb.example.org)
Stack C -- Creates Route53 ALIAS records aaa.example.org and bbb.example.org in a single hosted zone, needs the distribution domain names as input

The challenge is that account A is using a different profile than account B in ~/.aws/credentials so AWS CDK would need to access both profiles to retrieve the distribution domain names.

In realation aws#49 The action’s service roles is a role which will be assumed by pipeline during execution of this action. The pipeline action’s service role can be used to perform more advanced configuration, when i.e. elevation of permissions is required, or when fine grained access control may be required. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-codepipeline-pipeline-stages-actions.html This commit is motivated by enabling cross-account deployments, for which service role’s will be used as jump role to assume one used by Cloud Formation in target account.

In realation to aws#49 The action’s service roles is a role which will be assumed by pipeline during execution of this action. The pipeline action’s service role can be used to perform more advanced configuration, when i.e. elevation of permissions is required, or when fine grained access control may be required. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-codepipeline-pipeline-stages-actions.html This commit is motivated by enabling cross-account deployments, for which service role’s will be used as jump role to assume one used by Cloud Formation in target account.

In realation to aws#49 The action’s service roles is a role which will be assumed by pipeline during execution of this action. The pipeline action’s service role can be used to perform more advanced configuration, when i.e. elevation of permissions is required, or when fine grained access control may be required. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-codepipeline-pipeline-stages-actions.html This commit is motivated by enabling cross-account deployments, for which service role will be used as jump role to assume one used by Cloud Formation in target account.

aripalo · 2019-01-16T11:00:30Z

Use case:

I just hit this limitation by doing:

const primaryApi = new ApiStack(app, 'ApiStackIreland', { env: { region: 'eu-west-1' } }); // which has public readonly apiUrl: string;

new EdgeStack(app, 'EdgeStack', {
  primaryApiUrl: primaryApi.apiUrl,
  env: { region: 'us-east-1' }
});

And received this error:

node_modules/@aws-cdk/cdk/lib/cloudformation/cfn-tokens.js:65
            throw new Error('Can only reference cross stacks in the same region and account.');

Having "EdgeStack" in us-east-1 makes sense as it is CloudFront's "main region" and is a must as one has to define ACM certificates and Lambda@Edge there.

This currently is somewhat blocking.

Is there any workaround currently?

rix0rrr · 2019-01-16T14:33:30Z

There is not, unfortunately. You're going to have to thread the API URL across regions manually. That means:

Deploy the ApiStackIreland first. Note the API URL (maybe make it an Output so that it's easy to get to).
Deploy the EdgeStack and plug in the value of API URL from the first step.

There are a number of ways you could do this, and honestly I'm not sure which one to recommend.

I guess a pragmatic one would be to use this.node.getContext() and pass the value on the command-line when deploying the second stack (using the -c KEY=VALUE fflag), or put it into cdk.json under the context: {} section.

rix0rrr · 2019-01-16T14:34:28Z

You could do it automatically using a Lambda-backed Custom Resource in EdgeStack that would make a cross-region call to look up the API URL as well, but that's a little more work.

In realation to aws#49 The action’s service roles is a role which will be assumed by pipeline during execution of this action. The pipeline action’s service role can be used to perform more advanced configuration, when i.e. elevation of permissions is required, or when fine grained access control may be required. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-codepipeline-pipeline-stages-actions.html This commit is motivated by enabling cross-account deployments, for which service role will be used as jump role to assume one used by Cloud Formation in target account.

briannab12 · 2019-03-12T22:19:56Z

@skinny85 Changes have been made since July. cp.CodeCommitSource no longer exists. Is there still a way to set the role arn for a PipelineSourceAction? I am accessing a repo in a different AWS account.

skinny85 · 2019-03-12T23:15:59Z

Hey @briannab12 ,

currently, there is no way to do that. If you're using TypeScript, the following hack should work:

(sourceAction as any)['role'] = yourRole;

We're working on this experience as we speak, and we hope to make it awesome - see the #1924 PR for details.

Thanks,
Adam

briannab12 · 2019-03-13T13:40:17Z

(sourceAction as any)['role'] = yourRole;
This worked, thank you.

Here are the specifics of my circumstances:

        const RepoBranch = new cdk.Parameter(this, 'RepoBranch', {
		type: 'String',
		description: 'The branch name for the CodeCommit API repository.'
	});

        const sourceStage = pipeline.addStage('Source');

	const repo = cc.Repository.import(this, envNameParam+'RepoRef', {
		repositoryName: sourceRepoNameParam
	});
	
	const sourceActionApiRepo = new cc.PipelineSourceAction(this, 'SourceActionApiRepo', {
		stage: sourceStage,
		repository: repo,
		branch: RepoBranch.ref
	});

        const crossAccountRole = iam.Role.import(this, envNameParam+'CrossAccountRoleImport',{
		roleArn: crossAccountRoleArnCodeCommitParam
	});
	
	//Hacky was of setting source action ARN. Setting it is not currently supported by cdk. 
	(sourceActionApiRepo as any)['role'] = crossAccountRole;

windlessuser · 2019-08-02T17:24:50Z

Use case: Route53 hostname records in a shared hosted zone

Stack A -- defines a CloudFront distribution in AWS account A (aaa.example.org)

Stack B -- defines a CloudFront distribution in AWS account B (bbb.example.org)

Stack C -- Creates Route53 ALIAS records aaa.example.org and bbb.example.org in a single hosted zone, needs the distribution domain names as input

The challenge is that account A is using a different profile than account B in ~/.aws/credentials so AWS CDK would need to access both profiles to retrieve the distribution domain names.

@kennu did you find a workaround?

saltman424 · 2020-02-13T20:14:42Z

Use case: Route53 hostname records in a shared hosted zone

Stack A -- defines a CloudFront distribution in AWS account A (aaa.example.org)

Stack B -- defines a CloudFront distribution in AWS account B (bbb.example.org)

Stack C -- Creates Route53 ALIAS records aaa.example.org and bbb.example.org in a single hosted zone, needs the distribution domain names as input

The challenge is that account A is using a different profile than account B in ~/.aws/credentials so AWS CDK would need to access both profiles to retrieve the distribution domain names.

@kennu did you find a workaround?

@eladb Since this issue is now closed, do we have a solution for the above use case?

I am trying something similar: having a public hosted zone in one account delegate to a public hosted zone in another account.

cshenrik · 2020-04-22T20:36:25Z

@eladb Can you offer some insights into why this issue was closed? A good generic solution would have taken CDK to the next level (IMHO).

OperationalFallacy · 2020-12-21T17:32:49Z

It looks like the project is trying to get around Cloudformation's native limitation: it can't read outputs from another account/region. Having this functionality in Cloudformation would be a perfect world.

Meanwhile, cdk can deal with the env variable, contexts, config files - why not make stacks outputs automatically available for subsequent stages in a known artifact?

skinny85 · 2020-12-23T18:17:23Z

@OperationalFallacy I agree, and we have a feature request already for that functionality: #8566 .

Thanks,
Adam

alehatsman · 2021-03-05T12:56:03Z

Use case: Route53 hostname records in a shared hosted zone

Stack A -- defines a CloudFront distribution in AWS account A (aaa.example.org)

Stack B -- defines a CloudFront distribution in AWS account B (bbb.example.org)

Stack C -- Creates Route53 ALIAS records aaa.example.org and bbb.example.org in a single hosted zone, needs the distribution domain names as input

The challenge is that account A is using a different profile than account B in ~/.aws/credentials so AWS CDK would need to access both profiles to retrieve the distribution domain names.

@kennu did you find a workaround?

@eladb Since this issue is now closed, do we have a solution for the above use case?

I am trying something similar: having a public hosted zone in one account delegate to a public hosted zone in another account.

I am having a similar problem currently, has anyone tried to solve it using custom resources and cdk? A custom resource that calls lambda which is able to access the parent host zone and add records from stack A and B?

OperationalFallacy · 2021-03-05T13:18:02Z

Cdk has now a construct to add records with lambda, and it works well with pipelines

https://github.com/OperationalFallacy/CdkRouter53/pull/1/files#diff-28e171489876432b7aa1c09d7d1a446d777028d46e763081600b19ead74c9111R25

alehatsman · 2021-03-05T16:07:48Z

Cdk has now a construct to add records with lambda, and it works well with pipelines

https://github.com/OperationalFallacy/CdkRouter53/pull/1/files#diff-28e171489876432b7aa1c09d7d1a446d777028d46e763081600b19ead74c9111R25

Thank you for your reply! It works well!

rix0rrr added the design label Jun 22, 2018

rix0rrr mentioned this issue Jul 6, 2018

iam: Grants only work on IAM objects, not arbitrary principals #236

Closed

eladb mentioned this issue Jul 12, 2018

CloudWatch Logs: new library #307

Merged

skinny85 mentioned this issue Sep 10, 2018

Allow specifying runOrder and roleArn for CodePipeline Actions #686

Closed

fulghum added the post-GA label Sep 20, 2018

eladb mentioned this issue Oct 14, 2018

L2s: return a concrete physical name/ARN instead of token when possible #928

Closed

rix0rrr mentioned this issue Dec 11, 2018

In-app/cross-stack references #1324

Closed

srchase added needs-design and removed design labels Jan 3, 2019

rix0rrr mentioned this issue Feb 20, 2019

Feature Request: Expose AWS SDK to client code #1796

Closed

RomainMuller removed the post-GA label Jul 23, 2019

eladb closed this as completed Aug 27, 2019

amwill04 mentioned this issue Sep 18, 2019

Crossacount role for pipeline issue #4125

Closed

amhammett mentioned this issue Sep 10, 2020

[aws-codepipeline-actions] codebuild cross-region build actions #10279

Closed

2 tasks

iDVB mentioned this issue Jan 29, 2021

CDK Construct serverless-nextjs/serverless-next.js#878

Merged

7 tasks

jpSimkins mentioned this issue Feb 17, 2021

(aws-sns): Tests fail with: Only root constructs may have an empty name #13121

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cross-region/account references #49

Cross-region/account references #49

rix0rrr commented Jun 6, 2018

rix0rrr commented Jun 28, 2018

rix0rrr commented Jul 4, 2018

skinny85 commented Jul 6, 2018 •

edited

Loading

rix0rrr commented Jul 6, 2018

rix0rrr commented Jul 6, 2018

eladb commented Jul 12, 2018

rix0rrr commented Jul 12, 2018

eladb commented Jul 12, 2018

kennu commented Sep 23, 2018

aripalo commented Jan 16, 2019

rix0rrr commented Jan 16, 2019

rix0rrr commented Jan 16, 2019

briannab12 commented Mar 12, 2019 •

edited

Loading

skinny85 commented Mar 12, 2019

briannab12 commented Mar 13, 2019 •

edited

Loading

windlessuser commented Aug 2, 2019

saltman424 commented Feb 13, 2020

cshenrik commented Apr 22, 2020

OperationalFallacy commented Dec 21, 2020

skinny85 commented Dec 23, 2020

alehatsman commented Mar 5, 2021

OperationalFallacy commented Mar 5, 2021

alehatsman commented Mar 5, 2021

Cross-region/account references #49

Cross-region/account references #49

Comments

rix0rrr commented Jun 6, 2018

rix0rrr commented Jun 28, 2018

rix0rrr commented Jul 4, 2018

skinny85 commented Jul 6, 2018 • edited Loading

rix0rrr commented Jul 6, 2018

rix0rrr commented Jul 6, 2018

eladb commented Jul 12, 2018

rix0rrr commented Jul 12, 2018

eladb commented Jul 12, 2018

kennu commented Sep 23, 2018

aripalo commented Jan 16, 2019

rix0rrr commented Jan 16, 2019

rix0rrr commented Jan 16, 2019

briannab12 commented Mar 12, 2019 • edited Loading

skinny85 commented Mar 12, 2019

briannab12 commented Mar 13, 2019 • edited Loading

windlessuser commented Aug 2, 2019

saltman424 commented Feb 13, 2020

cshenrik commented Apr 22, 2020

OperationalFallacy commented Dec 21, 2020

skinny85 commented Dec 23, 2020

alehatsman commented Mar 5, 2021

OperationalFallacy commented Mar 5, 2021

alehatsman commented Mar 5, 2021

skinny85 commented Jul 6, 2018 •

edited

Loading

briannab12 commented Mar 12, 2019 •

edited

Loading

briannab12 commented Mar 13, 2019 •

edited

Loading