(vpc): context lookup does not occur/complete before vpc is referenced in other constructs

[peterwoodworth]

Describe the bug

This is a bit of a weird bug. If I try to lookup a vpc, and then try to select subnets based on that vpc, it will fail under certain conditions. When I try to use subnetFilters when selecting subnets following a lookup in my code, I will always get the error Cluster requires at least 2 subnets, got 0 before cdk.context.json is created. However if I instead try to filter by subnetType instead of with subnetFilters, a lookup will be successfully performed and the code will run normally.

Expected Behavior

For context lookup to occur before the vpc is referenced elsewhere

Current Behavior

Lookup does not complete and errors occur due to lack of subnets found

Reproduction Steps

Here's a minimal reproduction stack. Comment out subnetFilter and remove the comment on subnetType to create cdk.context.json. Once this file is created, you can remove the comment on subnetFilter` and successfully filter subnets.

    const vpc = ec2.Vpc.fromLookup(this, 'vpc',{
      vpcName: 'MyVpc'
    });

    const docDbCluster = new docdb.DatabaseCluster(this, 'DocDB', {
      masterUser: {
          username: 'myusername',
          excludeCharacters: ':*/?#[];%@"\'\\',
          secretName: `mysecretname`,
      },
      instanceType: ec2.InstanceType.of(
          ec2.InstanceClass.BURSTABLE3,
          ec2.InstanceSize.MEDIUM,
      ),
      vpcSubnets: {
          subnetFilters: [ 
            ec2.SubnetFilter.availabilityZones(["us-east-1a", "us-east-1b"]),
          ],
          // subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS,
      },
      vpc: vpc,
    });

Possible Solution

Lookup should fully complete before code referencing that lookup is ran. However, I'm not immediately sure why this is only occurring conditionally based on how subnets are selected from a looked up Vpc.

Additional Information/Context

May be related to #21690

CDK CLI Version

2.41.0

Framework Version

No response

Node.js Version

16

OS

mac

Language

Typescript

Language Version

No response

Other information

No response

[arewa]

Hi @peterwoodworth, I made an investigation and seems I found out why it occurs but don't know at the moment how to properly fix it.

In a nutshell, it depends on in which module the error occurs:

• when subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS is used then the error occurs in the ec2 module at this line
  

  aws-cdk/packages/@aws-cdk/aws-ec2/lib/vpc.ts

  Line 623 in aedc888

  throw new Error(`There are no '${subnetType}' subnet groups in this VPC. Available types: ${availableTypes}`);

• when the availability zones filter is applied, subnetType fallback to SubnetType.Public and pass the check above most likely because vpc has some public subnets by default. But then the error occurs in docdb module at this line:
  

  aws-cdk/packages/@aws-cdk/aws-docdb/lib/cluster.ts

  Line 399 in 74318c7

  throw new Error(`Cluster requires at least 2 subnets, got ${subnetIds.length}`);

LookedUpVpc has a context provider for context values and it has a fallback to add missing context values in case of provider error (lines 99 - 108):

aws-cdk/packages/@aws-cdk/core/lib/context-provider.ts

Line 99 in 74318c7

const providerError = extractProviderError(value);

But when the error happens at docdb it seems to be that vpc missing values remain unreported and the synthesizer doesn't save the context to the file:

aws-cdk/packages/aws-cdk/lib/api/cxapp/cloud-executable.ts

Line 81 in fedd51b

if (assembly.manifest.missing && assembly.manifest.missing.length > 0) {

[peterwoodworth]

Thanks for posting your findings @arewa, this is a big help!

[peterwoodworth]

Another instance of this occurring here

[RogerWilco80]

Hi @peterwoodworth, thank you for finding this one. I'm experiencing it too in CDK V2.80.0.

Are there any known workarounds for this issue? I'll try filtering by subnetType as you mentioned in your first post and see if it'll suit and how far I get.

[RogerWilco80]

Hi @peterwoodworth, Could you help please, I'm very stuck on this one.

I have the same issue where cdk.context.json isn't getting populated with the AZs for my env. The difference with me is I'm trying to select a subnet to use to create an EC2 instance.
I've tried using vpc.select_subnets(subnet_type=ec2.SubnetType.PUBLIC) and I get back ['dummy1a', 'dummy1b', 'dummy1c']. Then if I try and narrow it down to one subnet when creating the EC2 instance:

master_db=ec2.CfnInstance(
            self, "PSQL-Server",
            availability_zone="ap-southeast-2a",
            disable_api_termination=True,
            image_id=master_db_ami.image_id,
            instance_type="t3a.large",
            key_name=environment.ssh_keys[environment.deployEnv],
            private_dns_name_options=private_dns_name_opts,
            propagate_tags_to_volume_on_creation=True,
            subnet_id=vpc.select_subnets(
                availability_zones=["ap-southeast-2a"], 
                one_per_az=True, 
                subnet_type=ec2.SubnetType.PRIVATE_WITH_EGRESS
            ).subnet_ids[0],
            tags=[CfnTag(key="Name",value="PSQL-Server"),
                CfnTag(key="Service",value="PostgreSQL")                
            ]

I get an IndexError: list index out of range.

doing `print(self.availability_zones)' returns the expected result, but doesn't cache it in the context file.

Thank you in advance and let me know if you want me to create a seperate ticket for this.
Scott

[RogerWilco80]

Ok, so I re-read ticket 23077 again, then realised that even just having the code in app.py for the stack that creates the EC2 instance, even though I'm specifying a different stack when running cdk synth, is enough to generate the error. I commented out the stacks that create the EC2 instance and was able to synth (just creating the VPC was enough) to populate the cdl.context.json and now it's working enough to keep going.

[andreprawira]

@peterwoodworth i'm facing an issue similar to this and im wondering if you know how to solve it. So we have 2 stack, 1 deploys EKS cluster and the other deploys DocumentDB

eks stack

self.cluster = eks.Cluster(
            self,
            "eks-cluster",
            version=eks.KubernetesVersion.V1_28,
            kubectl_layer=lambda_layer_kubectl_v28.KubectlV28Layer(
                self, "kubectl-layer"
            ),
            place_cluster_handler_in_vpc=True,
            cluster_name=f"{props.customer}-eks-cluster",
            default_capacity_instance=ec2.InstanceType(props.worker_node_instance_type),
            default_capacity=2,
            alb_controller=eks.AlbControllerOptions(
                version=eks.AlbControllerVersion.V2_6_2
            ),
            vpc=ec2.Vpc.from_lookup(
                self, "vpc-lookup", vpc_name=f"{props.customer}-{props.region}/vpc"
            ),
            vpc_subnets=[ec2.SubnetSelection(subnet_group_name="application")],
        )

docb stack

cluster = docdb.DatabaseCluster(
            self,
            "database",
            master_user=docdb.Login(
                username="myuser",  # cant use'admin' because it is reserved by DocumentDB
                exclude_characters="!@#$%^&*()`~,}{[]_=+'?\/|<>:;-" + '"',
                secret_name=f"{props.customer}-documentdb-secrets",
            ),
            instance_type=ec2.InstanceType(
                instance_type_identifier=props.documentdb_instance_type
            ),
            vpc=ec2.Vpc.from_lookup(
                self, "vpc-lookup", vpc_name=f"{props.customer}-{props.region}/vpc"
            ),
            vpc_subnets=ec2.SubnetSelection(subnet_group_name="data"),
            db_cluster_name=f"{props.customer}-cluster",
            removal_policy=cdk.RemovalPolicy.DESTROY,
        )

We are importing the VPC and choosing the subnet based on the subnet group names, if we do a cdk synth, we will get RuntimeError: Cluster requires at least 2 subnets, got 0 error for the docdb stack and RuntimeError: Cannot place cluster handler in the VPC since no private subnets could be selected for the eks cluster stack.

We also found out that the cdk.context.json where it should store the information related to the VPC we look up is never generated.

But, if we deploy another postgres stack even using the same method to lookup the VPC and subnets, it works and cdk.context.json gets generated correctly

postgres stack

         domain = opensearchservice.Domain(
            self,
            "domain",
            version=opensearchservice.EngineVersion.OPENSEARCH_2_11,
            capacity=opensearchservice.CapacityConfig(master_nodes=3, data_nodes=3),
            ebs=opensearchservice.EbsOptions(
                volume_size=20, volume_type=ec2.EbsDeviceVolumeType.GP3
            ),
            zone_awareness=opensearchservice.ZoneAwarenessConfig(
                availability_zone_count=3
            ),
            logging=opensearchservice.LoggingOptions(
                slow_search_log_enabled=True,
                app_log_enabled=True,
                slow_index_log_enabled=True,
            ),
            domain_name=f"{props.customer}-{props.region}-domain",
            vpc=ec2.Vpc.from_lookup(
                self, "vpc-lookup", vpc_name=f"{props.customer}-{props.region}/vpc"
            ),
            vpc_subnets=[ec2.SubnetSelection(subnet_group_name="application")],
            removal_policy=RemovalPolicy.DESTROY,
        )

My issue is also related to this

[pahud]

RuntimeError: Cluster requires at least 2 subnets, got 0 error for the docdb stack and RuntimeError: Cannot place cluster handler in the VPC since no private subnets could be selected for the eks cluster stack.

This is generally because CDK VPC construct can't find the subnet IDs based on the passed subnet selection, which is

ec2.SubnetSelection(subnet_group_name="application")

If the cdk.context.json is not available in your deploying machine, the VPC construct would not be able to filter the subnet group name like. I would recommend using other subnet filters, for example: #28146 (comment)

Also, you generally don't have to do this

place_cluster_handler_in_vpc=True,

Is there any reason you need to enable this prop?

[andreprawira]

@pahud

Is there any reason you need to enable this prop?

Because we want the cluster to be placed in the private subnets, at least thats what we interpret it for.

If the cdk.context.json is not available in your deploying machine, the VPC construct would not be able to filter the subnet group name like. I would recommend using other subnet filters, for example: #28146 (comment)

Unfortunately, that will not work. We have 3 types of subnets:

• subnet group name public type public
• subnet group name data type isolated
• subnet group name application type isolated

If we use the alternative you mentioned, we will be using both subnet groups data and application for the eks and docdb stacks which we dont want, we want the eks to only use application subnet group and docdb to only use data subnet group

This is generally because CDK VPC construct can't find the subnet IDs based on the passed subnet selection

This still does not answer our question, why is it that even though in eks, docdb, and postgres stacks we use same method to lookup the VPC, we'd get an error when we cdk synth eks and docdb stacks, but if we comment them out and let the postgres stack be and cdk synth, the cdk.context.json gets generated succesfuly??

In other words,

vpc=ec2.Vpc.from_lookup(
                self, "vpc-lookup", vpc_name=f"{props.customer}-{props.region}/vpc"
            ),

vpc_subnets=ec2.SubnetSelection(subnet_group_name="data"),

or

vpc_subnets=ec2.SubnetSelection(subnet_group_name="application"),

only works if we use it in postgres stack, not in eks nor docdb, why is that?