[@aws-cdk/pipelines] failed to import a vpc from a stack into application stage #10095

enricopesce · 2020-09-01T14:18:45Z

A stack inside the CodePipeline application stage doesn't recognize an existing VPC

Reproduction Steps

export class ApplicationStage extends cdk.Stage {
    public readonly urlOutput: cdk.CfnOutput
    constructor(scope: cdk.Construct, id: string, props?: cdk.StageProps) {
        super(scope, id, props)
        const service = new ApplicationStack(this, id, config.VPC_NAME, props)
        this.urlOutput = service.urlOutput
    }
} 

export class ApplicationStack extends cdk.Stack {
	public readonly urlOutput: cdk.CfnOutput;

	constructor(scope: cdk.Construct, id: string, vpc_name: string, props?: cdk.StackProps) {
		super(scope, id, props)

		console.log("VPC NAME: ")
		console.log(JSON.stringify(vpc_name))
		console.log("PROPS: ")
		console.log(JSON.stringify(props))

		const vpc = ec2.Vpc.fromLookup(this, "vpc", { vpcName: vpc_name })

		console.log("VPC IMPORTED ")
		console.log(vpc)

		const sg = new ec2.SecurityGroup(this, 'sg', {
			vpc: vpc
		})

	}
}

What did you expect to happen?

I'm expecting the creation of the security group referenced to the existing VPC

What actually happened?

CloudFromation error:

The vpc ID 'vpc-12345' does not exist (Service: AmazonEC2; Status Code: 400; Error Code: InvalidVpcID.NotFound; Request ID: 1e003787-3b4f-4d13-8033-48fb609d4447; Proxy: null)

The console logs added for debugging inside the stack from the synth stage:

34 | VPC NAME:
35 | "VPC-RD"
36 | PROPS:
37 | {"env":{"account":"946412081729","region":"eu-west-1"}}
38 | VPC IMPORTED
39 | LookedUpVpc {
40 | node: ConstructNode {
41 | host: [Circular],
42 | _actualNode: Node {
43 | host: [Circular],
44 | _locked: false,
45 | _aspects: [],
46 | _children: [Object],
47 | _context: {},
48 | _metadata: [],
49 | _dependencies: Set {},
50 | invokedAspects: [],
51 | id: 'vpc',
52 | scope: [ApplicationStack]
53 | }
54 | },
55 | stack: ApplicationStack {
56 | node: ConstructNode { host: [Circular], _actualNode: [Node] },
57 | _missingContext: [ [Object] ],
58 | _stackDependencies: {},
59 | templateOptions: {},
60 | _logicalIds: LogicalIDs { renames: {}, reverse: {} },
61 | account: '946412081729',
62 | region: 'eu-west-1',
63 | environment: 'aws://946412081729/eu-west-1',
64 | terminationProtection: undefined,
65 | _stackName: 'stg-stg',
66 | tags: TagManager {
67 | tags: Map {},
68 | priorities: Map {},
69 | initialTagPriority: 50,
70 | resourceTypeName: 'aws:cdk:stack',
71 | tagFormatter: KeyValueFormatter {},
72 | tagPropertyName: 'tags'
73 | },
74 | artifactId: 'awsomepipelinespipelinestgC171B70D',
75 | templateFile: 'awsomepipelinespipelinestgC171B70D.template.json',
76 | synthesizer: DefaultStackSynthesizer {
77 | props: {},
78 | files: {},
79 | dockerImages: {},
80 | _stack: [Circular],
81 | bucketName: 'cdk-hnb659fds-assets-946412081729-eu-west-1',
82 | repositoryName: 'cdk-hnb659fds-container-assets-946412081729-eu-west-1',
83 | _deployRoleArn: 'arn:${AWS::Partition}:iam::946412081729:role/cdk-hnb659fds-deploy-role-946412081729-eu-west-1',
84 | _cloudFormationExecutionRoleArn: 'arn:${AWS::Partition}:iam::946412081729:role/cdk-hnb659fds-cfn-exec-role-946412081729-eu-west-1',
85 | fileAssetPublishingRoleArn: 'arn:${AWS::Partition}:iam::946412081729:role/cdk-hnb659fds-file-publishing-role-946412081729-eu-west-1',
86 | imageAssetPublishingRoleArn: 'arn:${AWS::Partition}:iam::946412081729:role/cdk-hnb659fds-image-publishing-role-946412081729-eu-west-1',
87 | _kmsKeyArnExportName: 'CdkBootstrap-hnb659fds-FileAssetKeyArn'
88 | },
89 | [Symbol(@aws-cdk/core.DependableTrait)]: { dependencyRoots: [Array] }
90 | },
91 | env: { account: '946412081729', region: 'eu-west-1' },
92 | _physicalName: undefined,
93 | _allowCrossEnvironment: false,
94 | physicalName: '${Token[TOKEN.187]}',
95 | natDependencies: [],
96 | incompleteSubnetDefinition: true,
97 | internetConnectivityEstablished: ConcreteDependable {
98 | _dependencyRoots: [],
99 | [Symbol(@aws-cdk/core.DependableTrait)]: { dependencyRoots: [Getter] }
100 | },
101 | vpcId: 'vpc-12345',
102 | cidr: '1.2.3.4/5',
103 | _vpnGatewayId: undefined,
104 | availabilityZones: [ 'dummy1a', 'dummy1b' ],
105 | publicSubnets: [
106 | ImportedSubnet {
107 | node: [ConstructNode],
108 | stack: [ApplicationStack],
109 | env: [Object],
110 | _physicalName: undefined,
111 | _allowCrossEnvironment: false,
112 | physicalName: '${Token[TOKEN.188]}',
113 | internetConnectivityEstablished: [ConcreteDependable],
114 | _availabilityZone: 'dummy1a',
115 | subnetId: 's-12345',
116 | routeTable: [Object],
117 | [Symbol(@aws-cdk/core.DependableTrait)]: [Object]
118 | },
119 | ImportedSubnet {
120 | node: [ConstructNode],
121 | stack: [ApplicationStack],
122 | env: [Object],
123 | _physicalName: undefined,
124 | _allowCrossEnvironment: false,
125 | physicalName: '${Token[TOKEN.189]}',
126 | internetConnectivityEstablished: [ConcreteDependable],
127 | _availabilityZone: 'dummy1b',
128 | subnetId: 's-67890',
129 | routeTable: [Object],
130 | [Symbol(@aws-cdk/core.DependableTrait)]: [Object]
131 | }
132 | ],
133 | privateSubnets: [
134 | ImportedSubnet {
135 | node: [ConstructNode],
136 | stack: [ApplicationStack],
137 | env: [Object],
138 | _physicalName: undefined,
139 | _allowCrossEnvironment: false,
140 | physicalName: '${Token[TOKEN.190]}',
141 | internetConnectivityEstablished: [ConcreteDependable],
142 | _availabilityZone: 'dummy1a',
143 | subnetId: 'p-12345',
144 | routeTable: [Object],
145 | [Symbol(@aws-cdk/core.DependableTrait)]: [Object]
146 | },
147 | ImportedSubnet {
148 | node: [ConstructNode],
149 | stack: [ApplicationStack],
150 | env: [Object],
151 | _physicalName: undefined,
152 | _allowCrossEnvironment: false,
153 | physicalName: '${Token[TOKEN.191]}',
154 | internetConnectivityEstablished: [ConcreteDependable],
155 | _availabilityZone: 'dummy1b',
156 | subnetId: 'p-67890',
157 | routeTable: [Object],
158 | [Symbol(@aws-cdk/core.DependableTrait)]: [Object]
159 | }
160 | ],
161 | isolatedSubnets: [],
162 | [Symbol(@aws-cdk/core.DependableTrait)]: { dependencyRoots: [ [Circular] ] }
163 | }

Environment

CLI Version : 1.61.1 (build 347918f)
Framework Version: 1.60.0
OS : aws/codebuild/standard:4.0
*Language (Version): TypeScript (3.9.7)

Other

The full code is present on this repo https://github.com/enricopesce/AWSome-pipeline/tree/pipelines

This is 🐛 Bug Report

The text was updated successfully, but these errors were encountered:

jpSimkins · 2020-09-01T15:24:38Z

Could this be due to the quotes on the VPC name? I am using the vpcId and it works as expected.

    const vpc = Vpc.fromLookup(this, "existingVPC", {
      vpcId: "vpc-1234567890"
    });

In your console, you JSON.stringify(vpc_name) to show "VPC-RD", was there a reason for this? What is the output of vpc_name? Perhaps try using the same stringify when using the lookup.

enricopesce · 2020-09-02T12:11:02Z

Usually, I use the name is it easy to remember :) the same code in a manual stack deployed works correctly, the console log prints the name of the VPC in the same way with the console.log(vpc_name) I have printed the value of the variable for a test.

enricopesce · 2020-09-02T12:29:47Z

I have tested the lookup with vpcid but in both cases, the synth phase produce a template with "vpc-12345"

{
  "Resources": {
    "sg29196201": {
      "Type": "AWS::EC2::SecurityGroup",
      "Properties": {
        "GroupDescription": "awsome-pipelines-pipeline/stg/stg/sg",
        "SecurityGroupEgress": [
          {
            "CidrIp": "0.0.0.0/0",
            "Description": "Allow all outbound traffic by default",
            "IpProtocol": "-1"
          }
        ],
        "VpcId": "vpc-12345"
      },
      "Metadata": {
        "aws:cdk:path": "awsome-pipelines-pipeline/stg/stg/sg/Resource"
      }
    }
  }
}

If the same class is deployed directly from CDK as standard stack WORKS 100% correctly if this class is deployed by aws-pipelines as a stage not!!

jguice · 2020-09-02T14:48:09Z

This looks like the known limitation of context queries not being support in pipelines (#8905 tracks the feature request).

enricopesce · 2020-09-03T09:43:27Z

Thank you @jguice is it a very big limitation!

rix0rrr · 2020-09-07T06:59:18Z

Don't print the result of a fromLookup(). On the first iteration it's not going to contain what you think it does.

sholtomaud · 2020-10-31T00:59:40Z

This issue shouldn't be closed just yet @rix0rrr - we need a formal workaround for the limitation of context queries before CDK Pipelines can be used for an enterprise production solution. Happy to test against an actual use case.

koups496 · 2020-11-03T17:02:43Z

I'm also looking for a workaround @rix0rrr

Is anyone using a workaround like this?

vpc = ec2.Vpc.from_vpc_attributes(self, 'VPC',
            vpc_id='vpc-xxxxxxxxx',
            availability_zones=["us-west-2a","us-west-2b","us-west-2c","us-west-2d"],
            private_subnet_ids=["subnet-xxxxxxxxxx", "subnet-xxxxxxxxxx", "subnet-xxxxxxxxxx", "subnet-xxxxxxxxxx"],

            )

I'm assuming most of us that are trying to use Vpc.fromLookup already know the vpc-id/az/subnet/etc information.

akuma12 · 2020-11-06T21:14:16Z

This is currently the only reason we're not using CDK Pipelines. Would love to see this issue resolved.

ayk33 · 2021-02-08T23:25:56Z

I'm also looking for a workaround @rix0rrr

Is anyone using a workaround like this?
vpc = ec2.Vpc.from_vpc_attributes(self, 'VPC',
            vpc_id='vpc-xxxxxxxxx',
            availability_zones=["us-west-2a","us-west-2b","us-west-2c","us-west-2d"],
            private_subnet_ids=["subnet-xxxxxxxxxx", "subnet-xxxxxxxxxx", "subnet-xxxxxxxxxx", "subnet-xxxxxxxxxx"],

            )
I'm assuming most of us that are trying to use Vpc.fromLookup already know the vpc-id/az/subnet/etc information.

So I tried using this solution but I end up with the warning:

[Warning at /cdk-dev01/cdk-dev01-vpc/PublicSubnet1] No routeTableId was provided to the subnet 'subnet-xxxx'. Attempting to read its .routeTable.routeTableId will return null/undefined. (More info: https://github.com/aws/aws-cdk/pull/3171)

[Warning at /cdk-dev01/cdk-dev01-vpc/PublicSubnet2] No routeTableId was provided to the subnet 'subnet-xxxx'. Attempting to read its .routeTable.routeTableId will return null/undefined. (More info: https://github.com/aws/aws-cdk/pull/3171)

[Warning at /cdk-dev01/cdk-dev01-vpc/PublicSubnet3] No routeTableId was provided to the subnet 'subnet-xxxx'. Attempting to read its .routeTable.routeTableId will return null/undefined. (More info: https://github.com/aws/aws-cdk/pull/3171)

[Warning at /cdk-dev01/cdk-dev01-vpc/PrivateSubnet1] No routeTableId was provided to the subnet 'subnet-xxxx'. Attempting to read its .routeTable.routeTableId will return null/undefined. (More info: https://github.com/aws/aws-cdk/pull/3171)

[Warning at /cdk-dev01/cdk-dev01-vpc/PrivateSubnet2] No routeTableId was provided to the subnet 'subnet-xxxx'. Attempting to read its .routeTable.routeTableId will return null/undefined. (More info: https://github.com/aws/aws-cdk/pull/3171)

[Warning at /cdk-dev01/cdk-dev01-vpc/PrivateSubnet3] No routeTableId was provided to the subnet 'subnet-xxxx'. Attempting to read its .routeTable.routeTableId will return null/undefined. (More info: https://github.com/aws/aws-cdk/pull/3171)

Anyone know how to fix this?

koups496 · 2021-02-08T23:48:19Z

Anyone know how to fix this?

@ayk33 Can you try defining the private_subnet_route_table_ids?

vpc = ec2.Vpc.from_vpc_attributes(self, 'VPC',
            vpc_id='vpc-xxxxxxxxx',
            availability_zones=["us-west-2a","us-west-2b","us-west-2c","us-west-2d"],
            private_subnet_ids=["subnet-xxxxxxxxxx", "subnet-xxxxxxxxxx", "subnet-xxxxxxxxxx", "subnet-xxxxxxxxxx"],
            private_subnet_route_table_ids=["id1","id2","id3","id4"]
            )

ayk33 · 2021-02-08T23:53:26Z

Yeah that works to remove the warning. Any idea why It can't just grab that information automatically though? If it's able to identify the VPC from the subnets I provide, I figured it should be able to get the route table ids?

koups496 · 2021-02-08T23:58:40Z

I think in this case it isn't doing a lookup so you define the attributes you need.

ayk33 · 2021-02-09T00:00:41Z

Makes sense. Do you know if it's possible to force it to do the lookup? Or will I need to just switch over to use something like this https://docs.aws.amazon.com/cdk/api/latest/python/aws_cdk.aws_ec2/VpcLookupOptions.html

koups496 · 2021-02-09T01:01:41Z

I have not tested that option. Give it a try.

ayk33 · 2021-02-09T18:55:53Z

For anyone who comes here in the future. Looks like aws_ec2.vpc.from_lookup() just works a lot better. Only needed to provide the vpc_id.

jmjava · 2021-02-10T22:10:04Z

@ayk33 I am trying from_lookup and seeing what is described early in this thread. I get "cannot find vpc-12345" Lookup works outside of pipeline for sure by name and id.

Does this actually work (defining the subnets?) in a pipeline?

vpc = ec2.Vpc.from_vpc_attributes(self, 'VPC',
vpc_id='vpc-xxxxxxxxx',
availability_zones=["us-west-2a","us-west-2b","us-west-2c","us-west-2d"],
private_subnet_ids=["subnet-xxxxxxxxxx", "subnet-xxxxxxxxxx", "subnet-xxxxxxxxxx", "subnet-xxxxxxxxxx"],
private_subnet_route_table_ids=["id1","id2","id3","id4"]
)

ayk33 · 2021-02-16T21:12:09Z

Yeah I got mine to work but just specifying the vpc_id. My guess would be check that you have the correct aws profile being used? Maybe it's checking the wrong account?

erikkinding · 2021-03-04T09:57:31Z

For what it's worth, here's the meat of what I ended up doing. This is used in a CDK app using a CDKPipeline. The lambda itself needs access to RDS, which is granted by associating the lambda with a pre-existing SecurityGroup.

from aws_cdk import core, \
    aws_codedeploy as codedeploy, \
    aws_lambda as lambda_, \
    aws_iam as iam, \
    aws_events, \
    aws_events_targets, \
    aws_ec2

import datetime


class MyLambdaStack(core.Stack):

    def __init__(self, app: core.App, id: str, **kwargs):
        super().__init__(app, id, **kwargs)

        # change me
        vpc_id = 'vpc-<your id>'
        sg_id = 'sg-<some existing security group with rds access>' 
        sub_net_ids = ["subnet-<some subnet id 1>", "subnet-<some subnet id 2>", "subnet-<some subnet id 3>"]
        azs = ["eu-west-1a", "eu-west-1b", "eu-west-1c"]
    
        # get vpc
        vpc = aws_ec2.Vpc.from_vpc_attributes(self, 'vpc', vpc_id=vpc_id, availability_zones=azs)
        sub_nets = [
		          aws_ec2.Subnet.from_subnet_attributes(self, f'subnet{idx}', subnet_id=sni)
		          for idx, sni in enumerate(sub_net_ids)
        ]
        sub_net_selection = aws_ec2.SubnetSelection(subnets=sub_nets)
    
        # Lambda SG
        sg = aws_ec2.SecurityGroup.from_security_group_id(self, 'lambda-rds', security_group_id=sg_id)
    
        func = lambda_.Function(self, "Lambda",
		                              code=self.lambda_code,
		                              handler="handler.lambda_handler",
		                              runtime=lambda_.Runtime.PYTHON_3_8,
		                              description="Function generated on {}".format(datetime.datetime.now()),
		                              vpc=vpc,
		                              timeout=core.Duration.seconds(10),
		                              vpc_subnets=sub_net_selection,
		                              security_groups=[sg]
		                              )


account = '<id without hyphen>'
region = 'eu-west-1'
env_eu = core.Environment(account=account, region=region)

app = core.App()
my_lambda_stack = MyLambdaStack(app, id="MyLambdaStack", env=env_eu)
app.synth()

oappicgi · 2021-07-09T10:21:12Z

At least with now relatively old version 1.89.0 of cdk and its packages vpclookup works. While it seems to be correct that cdk pipelines cannot use aws api to get vpc information with vpclookup inside pipeline you can still use it. It can read vpc data if it has been preloaded, so preload that information using cdk ls inside your project root and make sure cdk.context.json file is part of your commit before you push. Then cdk pipeline reads information contained in that file. This might not work with cross region nor cross account pipelines, but since I ran into same problem as people above and was bit surprised as it has worked for me on other project so I checked what i did differently and yup, after i generated cdk.context.json with cdk ls command and pushed it to repository it got correct vpc.

TL;DR: you can use vpclookup at least on same region/same account but you have to push cdk.context.json to repository which is generated by running cdk ls

enricopesce added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Sep 1, 2020

github-actions bot assigned rix0rrr Sep 1, 2020

github-actions bot added the @aws-cdk/pipelines CDK Pipelines library label Sep 1, 2020

rix0rrr closed this as completed Sep 7, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[@aws-cdk/pipelines] failed to import a vpc from a stack into application stage #10095

[@aws-cdk/pipelines] failed to import a vpc from a stack into application stage #10095

enricopesce commented Sep 1, 2020

jpSimkins commented Sep 1, 2020 •

edited

Loading

enricopesce commented Sep 2, 2020

enricopesce commented Sep 2, 2020 •

edited

Loading

jguice commented Sep 2, 2020

enricopesce commented Sep 3, 2020

rix0rrr commented Sep 7, 2020

sholtomaud commented Oct 31, 2020

koups496 commented Nov 3, 2020 •

edited

Loading

akuma12 commented Nov 6, 2020

ayk33 commented Feb 8, 2021 •

edited

Loading

koups496 commented Feb 8, 2021

ayk33 commented Feb 8, 2021

koups496 commented Feb 8, 2021

ayk33 commented Feb 9, 2021

koups496 commented Feb 9, 2021

ayk33 commented Feb 9, 2021

jmjava commented Feb 10, 2021

ayk33 commented Feb 16, 2021

erikkinding commented Mar 4, 2021 •

edited

Loading

oappicgi commented Jul 9, 2021

[@aws-cdk/pipelines] failed to import a vpc from a stack into application stage #10095

[@aws-cdk/pipelines] failed to import a vpc from a stack into application stage #10095

Comments

enricopesce commented Sep 1, 2020

Reproduction Steps

What did you expect to happen?

What actually happened?

Environment

Other

jpSimkins commented Sep 1, 2020 • edited Loading

enricopesce commented Sep 2, 2020

enricopesce commented Sep 2, 2020 • edited Loading

jguice commented Sep 2, 2020

enricopesce commented Sep 3, 2020

rix0rrr commented Sep 7, 2020

sholtomaud commented Oct 31, 2020

koups496 commented Nov 3, 2020 • edited Loading

akuma12 commented Nov 6, 2020

ayk33 commented Feb 8, 2021 • edited Loading

koups496 commented Feb 8, 2021

ayk33 commented Feb 8, 2021

koups496 commented Feb 8, 2021

ayk33 commented Feb 9, 2021

koups496 commented Feb 9, 2021

ayk33 commented Feb 9, 2021

jmjava commented Feb 10, 2021

ayk33 commented Feb 16, 2021

erikkinding commented Mar 4, 2021 • edited Loading

oappicgi commented Jul 9, 2021

jpSimkins commented Sep 1, 2020 •

edited

Loading

enricopesce commented Sep 2, 2020 •

edited

Loading

koups496 commented Nov 3, 2020 •

edited

Loading

ayk33 commented Feb 8, 2021 •

edited

Loading

erikkinding commented Mar 4, 2021 •

edited

Loading