aws_ec2: Cannot replace an instance when a private IP is set

[yuvashankar]

Describe the bug

It is possible to deploy an AWS EC2 Instance with a private IPv4 address (for example: 10.0.0.5). When we try to update the stack however, and the instance needs to be replaced, we find that the deployment fails with the error:

Resource handler returned message: "Address <private ip>  is in use. ...<snip>

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

The old instance is destroyed, the private IP is relinquished, and a new instance takes its place with the same IP address.

Current Behavior

The first cdk deploy command succeeds, but subsequent deploy commands fail with the following error. In this example, we were using 10.0.0.5 as the static private IP.

11:00:24 a.m. | CREATE_FAILED        | AWS::EC2::Instance                    | testinstanceinstan...61da7d69c9172494c5
Resource handler returned message: "Address 10.0.0.5 is in use. (Service: Ec2, Status Code: 400, Request ID: 7dc158b7-794c-4fad-a61a-e8f2d33b0e13)" (RequestToken: f3080a75-38cb-e40f-acfa-4d4c390
2a5be, HandlerErrorCode: InvalidRequest)


 ❌  ec2-instance failed: Error: The stack named ec2-instance failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "Address 10.0.0.5 is in use. (Service: Ec2, Status Code: 400, Request ID: 7dc158b7-794c-4fad-a61a-e8f2d33b0e13)" (RequestToken: f3080a75-38cb-e40f-acfa-4d4c3902a5be, HandlerErrorCode: InvalidRequest)
    at FullCloudFormationDeployment.monitorDeployment (/nix/store/qkdadnxhfbx9np4rlx2rd10k722dqbln-aws-cdk-2.159.1/lib/node_modules/aws-cdk/lib/index.js:463:10567)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/nix/store/qkdadnxhfbx9np4rlx2rd10k722dqbln-aws-cdk-2.159.1/lib/node_modules/aws-cdk/lib/index.js:466:200334)
    at async /nix/store/qkdadnxhfbx9np4rlx2rd10k722dqbln-aws-cdk-2.159.1/lib/node_modules/aws-cdk/lib/index.js:466:181756

 ❌ Deployment failed: Error: The stack named ec2-instance failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "Address 10.0.0.5 is in use. (Service: Ec2, Status Code: 400, Request ID: 7dc158b7-794c-4fad-a61a-e8f2d33b0e13)" (RequestToken: f3080a75-38cb-e40f-acfa-4d4c3902a5be, HandlerErrorCode: InvalidRequest)
    at FullCloudFormationDeployment.monitorDeployment (/nix/store/qkdadnxhfbx9np4rlx2rd10k722dqbln-aws-cdk-2.159.1/lib/node_modules/aws-cdk/lib/index.js:463:10567)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/nix/store/qkdadnxhfbx9np4rlx2rd10k722dqbln-aws-cdk-2.159.1/lib/node_modules/aws-cdk/lib/index.js:466:200334)
    at async /nix/store/qkdadnxhfbx9np4rlx2rd10k722dqbln-aws-cdk-2.159.1/lib/node_modules/aws-cdk/lib/index.js:466:181756

Reproduction Steps

Initialize a python3 cdk app

cdk init app --language python

Add reproduction code

Populate the app.py file with the following code, it is esentially a simplified version of the ec2/instances example.

#!/usr/bin/env python3
import os.path
import datetime

from aws_cdk import aws_ec2 as ec2, App, Stack
from constructs import Construct

dirname = os.path.dirname(__file__)


class EC2InstanceStack(Stack):
    def __init__(self, scope: Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        # VPC
        vpc = ec2.Vpc(
            self,
            "VPC",
            enable_dns_hostnames=True,
            enable_dns_support=True,
            subnet_configuration=[
                ec2.SubnetConfiguration(
                    cidr_mask=22,
                    name="public",
                    subnet_type=ec2.SubnetType.PUBLIC
                ),
            ],
        )

        # Instances
        test_instance = ec2.Instance(
            self,
            "test-instance-instance",
            instance_name="test-instance",
            instance_type=ec2.InstanceType.of(
                ec2.InstanceClass.T3, ec2.InstanceSize.NANO
            ),
            machine_image=ec2.MachineImage.latest_amazon_linux2(),
            vpc=vpc,
            vpc_subnets=ec2.SubnetSelection(subnet_type=ec2.SubnetType.PUBLIC),
            # Set a private IP and enforce that it gets replaced on every deploy
            private_ip_address="10.0.0.5",
            user_data_causes_replacement=True,
        )

        # Create a instance user script that changes on every deploy
        current_time = str(datetime.datetime.now())

        user_data_string = f"""
echo {current_time}
"""
        test_instance.add_user_data(user_data_string)


app = App()
EC2InstanceStack(app, "ec2-instance")

app.synth()

Deploy once

• Run the necessary steps to initialize the python3 stack (e.g. installing the venv etc etc)
• Run cdk deploy this should succeed.

Deploy again

• Run deploy again, changing the private ip.
• This should fail when the cdk attempts to replace the instance.
• There may need to be a cdk diff step done here, I'm unsure if that is important to force the cdk to destroy and create a new instance.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.159.1 (build c66f4e3)

Framework Version

No response

Node.js Version

v20.17.0

OS

Ubuntu 22.04.5 LTS

Language

Python

Language Version

Python 3.12.5

Other information

Here's my shell.nix file if it is helpful, along with the requirements.txt file created by the cdk CLI.

shell.nix

{pkgs ? import <nixpkgs> {}}:
let 
    pythonEnv = pkgs.python3.withPackages(ps: []);
in
pkgs.mkShell {
    buildInputs = with pkgs; [nodePackages.aws-cdk nodejs];
    packages = [pythonEnv];
}

requirements.txt

aws-cdk-lib==2.159.0
constructs>=10.0.0,<11.0.0

[khushail]

Hi @yuvashankar , thanks for reaching out.

Let me re-iterate what you are trying to do here-

1. create an EC2 instance and assign private ip
2. deploy
3. change the instance and expecting to change the private ip ???

I am rather confused as to why you would want to change the private ip because AFAIK, the private IP once assigned , are associated with the instance and will never change during the lifetime of that instance.

Could you please clarify as to what you are trying to do here and throw more light on your usecase?
Thanks

[yuvashankar]

1. create an EC2 instance and assign private ip
2. deploy
3. change the instance and expecting to change the private ip ???

That's almost correct, the issue occurs when the following sequence of events happen:

1. create an EC2 instance and assign a private IP
2. Deploy
3. Don't change the private IP, but for a variety of reasons aws-cdk determines that the instance has to be replaced.
4. Re-Deploy
5. The instance with the private IP set fails.

The crux of the issue is that when we replace an instance with a private IP set, the deployment fails complaining that the private IP is already allocated.

I've used the user_data_causes_replacement when creating an ec2.Instance Resource as a method to coerce the failure to occur consistently. Did that answer the question @khushail