EIP: AddressLimitExceeded, nat gateways with count > than 2

[ljankowski]

When we try to allocate nat gateways per availability zone, when var.zone_count more than 2 as per below:

resource "aws_eip" "nat_eip" {
    count = "${var.zone_count}"
    vpc = true
}

resource "aws_nat_gateway" "nat_gw" {
    count = "${var.zone_count}"
    allocation_id = "${element(aws_eip.nat_eip.*.id, count.index)}"
    subnet_id = "${element(aws_subnet.dmz.*.id, count.index)}"
    depends_on = ["aws_internet_gateway.vpc"]
}

the apply process fails with:

* aws_eip.nat_eip.2: Error creating EIP: AddressLimitExceeded: The maximum number of addresses has been reached.
    status code: 400, request id:

The problem is with order of operations.
As we use count to create nat gateways, I suspect terraform treats it as a one resource (and not 3).
and tries to allocate all IPs before creating even single nat gateway. Snip from the plan:

+ module.vpc.aws_eip.nat_eip.2
    allocation_id:     "" => "<computed>"
    association_id:    "" => "<computed>"
    domain:            "" => "<computed>"
    instance:          "" => "<computed>"
    network_interface: "" => "<computed>"
    private_ip:        "" => "<computed>"
    public_ip:         "" => "<computed>"
    vpc:               "" => "1"

+ module.vpc.aws_nat_gateway.nat_gw.0
    allocation_id:        "" => "${element(aws_eip.nat_eip.*.id, count.index)}"
    network_interface_id: "" => "<computed>"
    private_ip:           "" => "<computed>"
    public_ip:            "" => "<computed>"
    subnet_id:            "" => "subnet-7aedef23"

+ module.vpc.aws_nat_gateway.nat_gw.1
    allocation_id:        "" => "${element(aws_eip.nat_eip.*.id, count.index)}"
    network_interface_id: "" => "<computed>"
    private_ip:           "" => "<computed>"
    public_ip:            "" => "<computed>"
    subnet_id:            "" => "subnet-04d71a60"

+ module.vpc.aws_nat_gateway.nat_gw.2
    allocation_id:        "" => "${element(aws_eip.nat_eip.*.id, count.index)}"
    network_interface_id: "" => "<computed>"
    private_ip:           "" => "<computed>"
    public_ip:            "" => "<computed>"
    subnet_id:            "" => "subnet-e48d7892"

AWS limits number of unallocated EIPs to 2 (there's a charge for all consecutive ones), thus process fails.

Is it possible to either:

• add an option to aws_eip resource to accept the charge and carry on;
• change the dependency graph to split the resource when count is being used. problem should disappear as soon as one of the other nat gateways get eip allocated
• use try & carry on methodology as suggested here: thinking: dependency graphs are hard, so just don't do them #1732

Thanks,
Lukasz

[stack72]

Hi @ljankowski

Are you sure about this AWS limits number of unallocated EIPs to 2?

By default, the EIP limit is 5 so you may have 3 IPs already allocated and then trying to create 3 more will give you that error

I have just been able to create 5 new EIPs in my AWS account without any issues

thanks

Paul

[ljankowski]

Hi Paul,

We have few other addresses associated with instances, but no unassociated ones. So that's not it.

I did some more testing and it looks like when you destroy environment and rebuild it with 3 AZs, then all works as you said.

The problem exists, when you build with 2 AZs and then change the count from 2 to 3, then do plan and apply. In such situation it fails consistently.

Thanks,
Lukasz

[tecnobrat]

Amazon only allows 5 EIPs per region, when in a VPC. This is both unassociated and associated ones. They do not list a limit anywhere for "unassociated" anywhere that I can find.

http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html

Its under Amazon Virtual Private Cloud (Amazon VPC) Limits and you can search for Elastic IP addresses per region for each AWS account

The exception is being thrown by Amazon, not by terraform. You can find the exception details here: http://docs.aws.amazon.com/AWSEC2/latest/APIReference/errors-overview.html

Which specifically states You've reached the limit on the number of Elastic IP addresses that you can allocate.

[stack72]

Thanks for the info on this @tecnobrat

I am going to close this out for now as it corroborates my theory on hitting the EIP limit. If you raise the limit and find the same issue, please re-comment on this

thanks

Paul

[ljankowski]

@tecnobrat actually I did some more tests and you are right about the limit.

I have 3 IPs allocated for other purpose on that account in that region.

When I use terraform to create vpc with 2 IPs, then all works fine.
When I increase the count to 3, then it fails with LimitExceeded error. Which rightly takes me above 5 IP limit.

However, there's probably bug in AWS.
When I create vpc in 3AZs (with 3IPs) all works fine, even when I have other 3IPs allocated. That takes me above the 5 limit without AWS noticing it. IMO because terraform runs everything in parallel, there must be a race condition in AWS. Which is actually funny as I am able to create more resources that I am allowed to.

I should probably report that to AWS.

[yyarmoshyk-informa]

Open the Service Quotas console at https://console.aws.amazon.com/servicequotas/, enter Amazon EC2 in the search field, and choose Amazon Elastic Compute Cloud (Amazon EC2). Enter IP in the search field.

From this page you can increase the limit of available EIPs. The changes are being applied during 30 minutes

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.