Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(core): automatic cross stack, cross region references (under feature flag) #22008

Merged
merged 37 commits into from
Oct 31, 2022
Merged

feat(core): automatic cross stack, cross region references (under feature flag) #22008

merged 37 commits into from
Oct 31, 2022

Conversation

corymhall
Copy link
Contributor

@corymhall corymhall commented Sep 12, 2022

This PR adds the ability to automatically create references in cross-region stacks. You can now do something like

const stack1 = new Stack(app, 'Stack1', { env: { region: 'us-east-1' } });
const cert = new certificatemanager.Certificate(stack1, 'Cert', {...});

const stack2 = new Stack(app, 'Stack2', { env: { region: 'us-east-2' } });
new cloudfront.Distribution(stack2, 'Distro', {
  certificate: cert,
})

The above is a good example of the motivation behind this feature. A CloudFront distribution is a global resource and can be created in a CloudFormation stack in any region. Other resources, like the ACM certificate, that need to be attached to the CloudFront distribution can only be created in us-east-1. Another example is the CloudFront.EdgeFunction where we use a support stack and a custom resource to lookup the value.

To accomplish this, I've added two new constructs ExportsWriter & ExportReader. These constructs create Lambda backed custom resources.

ExportWriter responsibilities

  • Create/Update SSM parameters in the target region for each export
    • Will first check to make sure that the export is not "imported" by the consuming stack. If it is, then it will not update the value. This is to mimic the behavior of CloudFormation stack exports.

ExportReader responsibilities

  • Tag/Untag parameter indicating whether the parameter has been "imported"
  • Delete all parameters if the stack is deleted.

I am currently using /cdk/exports/${consumingStackName}/ as the SSM path prefix to create all the exports.

Given the above example, this would create an output in stack1

{
  "ExportsWriteruseast2828FA26B": {                                                                                                                                                                                    
   "Type": "Custom::CrossRegionExportWriter",                                                                                                                                                                          
   "Properties": {                                                                                                                                                                                                     
    "ServiceToken": {                                                                                                                                                                                                  
     "Fn::GetAtt": [                                                                                                                                                                                                   
      "CustomCrossRegionExportWriterCustomResourceProviderHandlerD8786E8A",                                                                                                                                            
      "Arn"                                                                                                                                                                                                            
     ]                                                                                                                                                                                                                 
    },                                                                                                                                                                                                                 
    "Region": "us-east-2",                                                                                                                                                                                             
    "Exports": {                                                                                                                                                                                                       
     "/cdk/exports/East2Stack/East1Stackuseast1CertRefCert5C9FAEC135985652": {                                                                                                                                                                            
      "Ref": "Cert5C9FAEC1"                                                                                                                                                                                            
     }                                                                                                                                                                                                                 
    }                                                                                                                                                                                                                  
   }, 
}

And then an "import" in stack2 which is a dynamic ssm reference.

{
 "Resources": {
  "Distro87EBE6BA": {
   "Type": "AWS::CloudFront::Distribution",
   "Properties": {
    "DistributionConfig": {
     "ViewerCertificate": {
      "AcmCertificateArn": "{{resolve:ssm:/cdk/exports/East2Stack/East1Stackuseast1CertRefCert5C9FAEC135985652}}"
    }
   }
  }
 }
}

Currently this will create a single ExportsWriter per region, but we could potentially update this to just use a single ExportsWriter which can write exports to a list of regions.

Future extensions:

  • Could be updated to do cross account references as well
  • Could be used to implement general weak references

All Submissions:

Adding new Unconventional Dependencies:

  • This PR adds new unconventional dependencies following the process described here

New Features

  • Have you added the new feature to an integration test?
    • Did you use yarn integ to deploy the infrastructure and generate the snapshot (i.e. yarn integ without --dry-run)?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

This PR adds the ability to automatically create references in
cross-region stacks. You can now do something like

```ts
const stack1 = new Stack(app, 'Stack1', { env: { region: 'us-east-1' } });
const cert = new certificatemanager.Certificate(stack1, 'Cert', {...});

const stack2 = new Stack(app, 'Stack2', { env: { region: 'us-east-2' } });
new cloudfront.Distribution(stack2, 'Distro', {
  certificate: cert,
})
```

To accomplish this, I've added a new construct `ExportsReader` which
creates a Lambda backed custom resource. This custom resource will
return all of the CloudFormation exports in a given region.

Given the above example, this would create an output in `stack1`

```json
{
  "Outputs": {
    "ExportsOutputRefCert5C9FAEC110AE5C6A": {
      "Value": {
        "Ref": "Cert5C9FAEC1"
      },
      "Export": {
        "Name": "East1:ExportsOutputRefCert5C9FAEC110AE5C6A"
      }
    }
  }
}
```

And then an "import" in `stack2` which references the `ExportReader`

```json
{
 "Resources": {
  "Distro87EBE6BA": {
   "Type": "AWS::CloudFront::Distribution",
   "Properties": {
    "DistributionConfig": {
     "ViewerCertificate": {
      "AcmCertificateArn": {
       "Fn::GetAtt": [
        "ExportsReaderuseast1D746CBDB",
        "East1:ExportsOutputRefCert5C9FAEC110AE5C6A"
       ]
      }
    }
   }
  }
 }
}
```

Currently this will create a single ExportsReader per region, but we
could potentially update this to just use a single ExportsReader which
can list exports from a list of regions.

Future extensions:
- Could be updated to do cross account references as well
- Could be used to implement weak references
@gitpod-io
Copy link

gitpod-io bot commented Sep 12, 2022

@mergify mergify bot added the contribution/core This is a PR that came from AWS. label Sep 12, 2022
@aws-cdk-automation aws-cdk-automation requested a review from a team September 12, 2022 19:01
@github-actions github-actions bot added the p2 label Sep 12, 2022
@corymhall corymhall marked this pull request as ready for review September 14, 2022 18:00
@@ -0,0 +1,65 @@
import { Queue, IQueue } from '@aws-cdk/aws-sqs';
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Put the integration test here since we can't have integration tests in core and there is a precedent for doing it this way.


const artifact = new Artifact();
const pipeline = new Pipeline(stack2, 'Pipeline', {
crossRegionReplicationBuckets: {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created this integration test based off of a common cross region scenario

### Creating an encrypted replication bucket
If you're passing a replication bucket created in a different stack,
like this:
```ts
// Passing a replication bucket created in a different stack.
const app = new App();
const replicationStack = new Stack(app, 'ReplicationStack', {
env: {
region: 'us-west-1',
},
});
const key = new kms.Key(replicationStack, 'ReplicationKey');
const replicationBucket = new s3.Bucket(replicationStack, 'ReplicationBucket', {
// like was said above - replication buckets need a set physical name
bucketName: PhysicalName.GENERATE_IF_NEEDED,
encryptionKey: key, // does not work!
});
// later...
new codepipeline.Pipeline(replicationStack, 'Pipeline', {
crossRegionReplicationBuckets: {
'us-west-1': replicationBucket,
},
});
```
When trying to encrypt it
(and note that if any of the cross-region actions happen to be cross-account as well,
the bucket *has to* be encrypted - otherwise the pipeline will fail at runtime),
you cannot use a key directly - KMS keys don't have physical names,
and so you can't reference them across environments.
In this case, you need to use an alias in place of the key when creating the bucket:
```ts
// Passing an encrypted replication bucket created in a different stack.
const app = new App();
const replicationStack = new Stack(app, 'ReplicationStack', {
env: {
region: 'us-west-1',
},
});
const key = new kms.Key(replicationStack, 'ReplicationKey');
const alias = new kms.Alias(replicationStack, 'ReplicationAlias', {
// aliasName is required
aliasName: PhysicalName.GENERATE_IF_NEEDED,
targetKey: key,
});
const replicationBucket = new s3.Bucket(replicationStack, 'ReplicationBucket', {
bucketName: PhysicalName.GENERATE_IF_NEEDED,
encryptionKey: alias,
});
```

@corymhall corymhall changed the title feat(core): automatic cross stack, cross region references feat(core): automatic cross stack, cross region references (under feature flag) Sep 15, 2022
@corymhall corymhall added pr-linter/exempt-readme The PR linter will not require README changes and removed pr-linter/exempt-readme The PR linter will not require README changes labels Sep 15, 2022
Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Pull Request Linter fails with the following errors:

❌ Features must contain a change to a README file.

PRs must pass status checks before we can provide a meaningful review.

@github-actions github-actions bot dismissed their stale review September 23, 2022 19:36

Pull Request updated. Dissmissing previous PRLinter Review.

Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Pull Request Linter fails with the following errors:

❌ Features must contain a change to a README file.

PRs must pass status checks before we can provide a meaningful review.

@github-actions github-actions bot dismissed their stale review September 23, 2022 19:38

Pull Request updated. Dissmissing previous PRLinter Review.

Comment on lines +9 to +15
const account = process.env.CDK_INTEG_ACCOUNT ?? process.env.CDK_DEFAULT_ACCOUNT;
const hostedZoneId = process.env.CDK_INTEG_HOSTED_ZONE_ID ?? process.env.HOSTED_ZONE_ID;
if (!hostedZoneId) throw new Error('For this test you must provide your own HostedZoneId as an env var "HOSTED_ZONE_ID"');
const hostedZoneName = process.env.CDK_INTEG_HOSTED_ZONE_NAME ?? process.env.HOSTED_ZONE_NAME;
if (!hostedZoneName) throw new Error('For this test you must provide your own HostedZoneName as an env var "HOSTED_ZONE_NAME"');
const domainName = process.env.CDK_INTEG_DOMAIN_NAME ?? process.env.DOMAIN_NAME;
if (!domainName) throw new Error('For this test you must provide your own Domain Name as an env var "DOMAIN_NAME"');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oof! Can we not create a zone as part of the test?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to have a valid public zone one way or another. We could create one in the test, but in order for it to be valid you would have to get the NS records and update the parent hosted zone.

Comment on lines 167 to 170
You can enable the feature flag `@aws-cdk/core:enableCrossRegionReferencesUsingCustomResources`
in order to access resources in a different stack _and_ region. With this feature flag
enabled it is possible to do something like creating a CloudFront distribution in `us-east-2` and
an ACM certificate in `us-east-1`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be in the CloudFront README (as well)

packages/@aws-cdk/core/README.md Show resolved Hide resolved
case 'Create':
console.info('Tagging SSM Parameter imports');
await addTags(ssm, imports, keyName);
return;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like we need to read the value THROUGH a { Fn::GetAtt } of this resource, to make sure the tagging happens before the value is consumed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking that we want to tag the resource after it is consumed. That way if it is not consumed (maybe the stack operation fails) we don't add the tags. Although I guess in that case we would rollback this resource as well which should remove the tags.

Also, if we read the value through this resource then in order for updates to be consumed then this resource needs to be updated. Right now I guess that is not a problem since we do not allow updates to exported values, but if we introduced weak references in the future that might be a problem.

Copy link
Contributor

@rix0rrr rix0rrr Oct 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alth ough I guess in that case we would rollback this resource as well which should remove the tags.

👆 this

If consuming and tagging are not connected, then we might have a window where a parameter is consumed and not tagged, which is dangerous because it might be replaced in that window.

Also, if we read the value through this resource then in order for updates to be consumed then this resource needs to be updated.

We could solve that by sending the SSM values in as Properties -- that way if the value changes, the CR is invalidated.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could solve that by sending the SSM values in as Properties -- that way if the value changes, the CR is invalidated.

genius!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok this is updated.

await putParameters(ssm, newExports);
return;
case 'Delete':
// consuming stack will delete parameters
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be safe to delete parameters that are not in use, right? I'm afraid we might leak them otherwise.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We would leak them until the consuming stack is deleted since the consuming stack will delete all of it's parameters (by path prefix). As it is we wouldn't have a list of all parameters that this resource has ever created, just what it had on the last update. We could store the state somewhere (another parameter?).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we would be able to delete the parameters from the producing stack if we only ever supported strong references (since the stack deployment will fail if you try to delete an imported parameter), but if we want to one day support weak references we need to "leak" the parameters from the producing stack.

Copy link
Contributor

@rix0rrr rix0rrr Oct 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We know which references are strong or weak beforehand, so we can apply different rules for them.

Right now, if the producing stack fails to create for some reason and rolls back (or the consumer fails to create) we never get to deploy the consumer and we will leak.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I reworked this. Since we want to fail the stack operation if we are trying to update/delete exports that are in use, i've moved the deletion to the ExportWriter. The ExportReader no longer deletes parameters, it will only add/remove tags.

@rix0rrr rix0rrr added the pr/do-not-merge This PR should not be merged at this time. label Oct 31, 2022
Copy link
Contributor

@rix0rrr rix0rrr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final change request: turn optInToCrossRegionReferences into crossRegionReferences ?

@corymhall corymhall removed the pr/do-not-merge This PR should not be merged at this time. label Oct 31, 2022
@mergify
Copy link
Contributor

mergify bot commented Oct 31, 2022

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@mergify
Copy link
Contributor

mergify bot commented Oct 31, 2022

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
  • Commit ID: 6197448
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mergify mergify bot merged commit f1b5497 into aws:main Oct 31, 2022
@mergify
Copy link
Contributor

mergify bot commented Oct 31, 2022

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@niebloomj
Copy link

Where does it say what the feature flag is named?

@kiernan
Copy link

kiernan commented Nov 11, 2022

This sounds great. Could something like this also help us with cross-stage dependencies?

@rittneje
Copy link

rittneje commented Jan 19, 2023

Will first check to make sure that the export is not "imported" by the consuming stack. If it is, then it will not update the value. This is to mimic the behavior of CloudFormation stack exports.

@corymhall Standard CloudFormation exports also cannot be deleted if they are in use. (If you try, the entire stack update/delete is immediately aborted.) However, since your approach models these exports as custom resources, that means they won't be deleted until the end of the deployment. Furthermore, CloudFormation basically ignores failures that occur during the UPDATE_COMPLETE_CLEANUP_IN_PROGRESS phase, meaning that the custom resource (and anything it manages) will just get leaked. Is this analysis correct?

On a related note, I think you may need something like Stack.export_value in order to address the "deadly embrace" issue that used to plague regular exports.

@DimitriosKay
Copy link

I take it this does not apply if the two cross-region stacks are deployed through separate CDK apps ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
contribution/core This is a PR that came from AWS. p2 pr-linter/exempt-readme The PR linter will not require README changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants