Large aws_waf_ipset resources cannot be deleted

[hashibot]

This issue was originally opened by @longwave as hashicorp/terraform#15146. It was migrated here as part of the provider split. The original body of the issue is below.

---

Terraform Version

Terraform v0.9.6

Affected Resource(s)

• aws_waf_ipset

Terraform Configuration Files

resource "aws_waf_ipset" "ip1" {
  name = "WAF-IP1"
}

in conjunction with the Lambda function from https://github.com/awslabs/aws-waf-sample/tree/master/waf-reputation-lists/lambda

Expected Behavior

A very large IP set that is no longer required should be deleted.

Actual Behavior

The IP set is not deleted and the following error is reported:

• aws_waf_ipset.ip1: Error updating IPSetDescriptors: Error Updating WAF IPSet: WAFLimitsExceededException: Operation would result in exceeding resource limits.

Steps to Reproduce

WAF IP sets can now contain 10,000 records but the AWS API only allows 1,000 to be created, updated or deleted in a single call.

This particular WAF IP set was built as empty with Terraform and then filled with the sample AWS Lambda function from https://github.com/awslabs/aws-waf-sample/tree/master/waf-reputation-lists/lambda, modified to support the new AWS limits via the comments in amazon-archives/aws-waf-sample#8. It was then no longer required so I removed the resource from the Terraform config and ran terraform apply which should have destroyed it, but I got the above error.

There is presumably a similar bug if you attempt to create an IP set with over 1,000 records directly in Terraform.

[willnewby]

I have the same issue, but updating an IPset with 9000 new entries.

Plan output:

. . .
ip_set_descriptors.#:                "0" => "9736"
. . .

Apply output:

Error applying plan:

1 error(s) occurred:

* aws_waf_ipset.ipset_11: 1 error(s) occurred:

* aws_waf_ipset.ipset_11: Error Updating WAF IPSet: Error Updating WAF IPSet: WAFLimitsExceededException: Operation would result in exceeding resource limits.
	status code: 400, request id: daa0c92b-6d79-11e7-b92a-0706c39b2685

[willfarrell]

So, you can create an ipset with up to 10,000 ips. However you can only add/remove 1000 at a time with the AWS API (see AWS support message below).

We should update this issue to request that aws_waf_ipset have a mechinism to create an empty ipset, then be recursive update with blocks of 1000 ips. The opposite would happen for delete; recursively remove ips in blocks of 1000, then delete when empty.

Related to: amazon-archives/aws-waf-sample#8

AWS support:

Thank you for providing me the github link that you are referring. It looks like that was posted back in March 2016. Back then there was a limit of 1,000 IP addresses per IPset condition. So, you had to combine multiple rules to support >1,000 IP addresses. However, in May 2017, AWS announced that it supports 10,000 CIDR or IP Address in a single IPSet condition in an AWS WAF Rule. So now, you do not have to combine multiple rules to achieve this. Please refer the attached links [1] [2]. 

Would you be able to provide me the output of the API call that you are making? It could be that you are hitting a limit, but limit could be the maximum number of updates per call. The API only allows 1,000 IPs to be added or removed with one call. But the over all IPset can contain 10,000 IP addresses, which would be sufficient for the Route53 health check IP address.

I am looking forward to your response. Meanwhile, if you have any questions around this please do let me know and I will be happy to help you.

[1] - https://aws.amazon.com/about-aws/whats-new/2017/05/limit-increase-for-ipset-conditions-for-aws-waf/

[2] - https://docs.aws.amazon.com/waf/latest/developerguide/web-acl-ip-conditions.html

[samm-git]

I can confirm this. Also very easy to reproduce with api tools:

aws waf update-ip-set --ip-set-id "<set>" --updates file://waf.json --change-token "<token>"

If waf.json contains < 1000 addresses it works, if > it is not. I can import 1600 addresses by splitting list to 2 and running command 2 times. I already requested information from AWS support to see if it is
aws bug. I will update this thread. Bug also can be changed to "deleted OR added" because it is same issue - any changelog > 1000 items is rejected.

[samm-git]

Actually it is documented in the api docs:

You can insert a maximum of 1000 addresses in a single request.

[bflad]

Bug fix pull request submitted: #5588

[bflad]

The fix for this has been merged into master and will release with version 1.33.0 of the AWS provider, likely later this week. 👍

[bflad]

This has been released in version 1.33.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!