No way to specify primary IP on aws_network_interface when multiple IPs in play

[pmcevoy]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.12.8
+ provider.aws v2.33.0

Affected Resource(s)

• aws_network_interface

Terraform Configuration Files

resource "aws_network_interface" "eni0_public" {
	count = length(local.zones)

	subnet_id         = data.aws_subnet.public[count.index].id
	security_groups   = [data.aws_security_group.fgt.id]
	source_dest_check = false
	private_ips       = [ 
		cidrhost(data.aws_subnet.public[count.index].cidr_block, 11),
		cidrhost(data.aws_subnet.public[count.index].cidr_block, 13)
	]
	description       = "${local.name_qualifier} ${count.index} port1 - External"
	tags = {
		Name = "${local.name_qualifier} ${count.index} port1 - External"
	}
}

Expected Behavior

In order to setup a HA Fortigate firewall in AWS, I need to create mulitple ENI with specific configuration, including fixed private IPs. I expect the above configration to create an ENI with a primary private IP of 10.85.36.11 (the first IP in the private_ips list) and secondary IP of 10.85.36.13.

Actual Behavior

In my Dev AWS account (with VPC CIDR 10.80.32.0/19), the above configuration creates an ENI with correct assignments (namely a primary of 10.80.36.11 and secondary of 10.80.36.13.

However in my Staging AWS enviroment (VPC CIDR of 10.85.32.0/19) the above configuration creates an ENI with incorrect assignments (namely a primary of 10.80.36.13 and secondary of 10.80.36.11).

In Dev, I can re-run the configuration multiple times and it always is correct, in Staging I have re-run the configuration multiple times and it is always incorrect

Important Factoids

It appears that it is possible to specify (undocumented) private_ip in addition to private_ips (I was hoping to be able to specify the primary, but that appears to be ignored

[pmcevoy]

See also resource: network_interface, private_ips is documented as a list, while it's a set

[pmcevoy]

I did some digging and there are multiple bugs and PRs related to this defect since 2017.. PLEASE PLEASE someone fix this.. it seems that the correct behaviour should be as detailed in this comment

Fixes #836 - correctly assign primary private IP for aws_network_instance when private_ips are specified

Sort private ips for an AWS instance in Terraform

resource/aws_network_interface: Use first of private_ips as primary

aws_network_interface.private_ips and aws_network_interface.private_ips_count act in an unexpected way

[steveh565]

I can confirm the same behavior on other virtual network appliances in AWS. In the same VPC I create subnets in two AZs, and deploy an instance into each AZ. The instance that deploys into AZ-a will always correctly assign primary and secondary private IPs to my NICs, however when I deploy an instance into AZ-b, that instance will always have primary and secondary IPs reversed.

example:

Create and attach bigip tmm network interfaces

resource "aws_network_interface" "az2_dmz_mgmt" {
depends_on = [aws_security_group.sg_ext_mgmt]
subnet_id = aws_subnet.az2_mgmt.id
private_ips = [var.az2_dmzF5.mgmt]
security_groups = [aws_security_group.sg_ext_mgmt.id]
}

resource "aws_network_interface" "az2_dmz_external" {
depends_on = [aws_security_group.sg_external]
subnet_id = aws_subnet.az2_dmzExt.id
private_ips = [var.az2_dmzF5.dmz_ext_self, var.az2_dmzF5.dmz_ext_vip]
security_groups = [aws_security_group.sg_external.id]
}

resource "aws_network_interface" "az2_dmz_internal" {
depends_on = [aws_security_group.sg_internal]
subnet_id = aws_subnet.az2_dmzInt.id
private_ips = [var.az2_dmzF5.dmz_int_self, var.az2_dmzF5.dmz_int_vip]
security_groups = [aws_security_group.sg_internal.id]
}

BigIP 2

resource "aws_instance" "az2_dmz_bigip" {
depends_on = [aws_subnet.az2_mgmt, aws_security_group.sg_ext_mgmt, aws_network_interface.az2_dmz_external, aws_network_interface.az2_dmz_internal, aws_network_interface.az2_dmz_mgmt]
ami = var.ami_f5image_name
instance_type = var.ami_dmz_f5instance_type
availability_zone = "${var.aws_region}b"
user_data = data.template_file.vm_onboard.rendered
key_name = "kp${var.tag_name}"
root_block_device {
delete_on_termination = true
}
network_interface {
device_index = 0
network_interface_id = aws_network_interface.az2_dmz_mgmt.id
}
network_interface {
device_index = 1
network_interface_id = aws_network_interface.az2_dmz_external.id
}
network_interface {
device_index = 2
network_interface_id = aws_network_interface.az2_dmz_internal.id
}
}

Despite the fact I order my declaration specifying the primary private IP first in the list, what I get in AWS EC2 console is the opposite of what I'm expecting (and require since the private secondary IPs are dependencies of other configurations):

12.12 is supposed to be a primary IP for eth1, and 12.112 is supposed to be secondary for eth1 but it's been reversed somehow.

I need a way to control this behavior. Please fix this issue.

[rick-masters]

This is a challenging problem with many issues involved. It took some time to unravel and create a potential solution. So, this comment is very long. You'd probably only want to read this if you really care about the issue of managing multiple IP addresses on an AWS network interface using terraform.

I should note up front that I am just some guy on the Internet trying to make a contribution. I only starting working on this project last week and I am not affiliated with Hashicorp, who ultimately decides what to do here. Normally I would collaborate before getting this far but I wanted to come with a proposal in hand that I knew would work and it actually took four attempts before I figured out a workable solution that I could live with. By that time it was mostly filled out so I just went ahead and finished it.

This comment is organized into these sections:

• How it Works Now
• Customer Problems
• Requirements
• Proposed Solution
• Specification
• Issues Resolved

How it Works Now

The describes the behavior of terraform 0.14.7 with AWS provider 3.29.0.

The existing IPv4 settings/exports are:

• private_ips
• private_ips_count
• private_ip

The existing IPv6 settings are:

• ipv6_addresses
• ipv6_address_count

private_ips

private_ips allows specifying a collection of private_ips to be configured on the interface. private_ips is considered to be an unordered set. Of course, in the config file the list of IPs appear in a sequential order but terraform does not care or remember what that order is. You can rearrange the order in the config file and terraform will not consider that to be a change.

Upon creation of an interface, Terraform picks the "primary" private IP from the list of private_ips using a consistent hash that produces the same primary IP as long as you have provided the same set of IPs. (It appends ";" to the address and uses https://github.com/hashicorp/terraform-plugin-sdk/blob/master/internal/helper/hashcode/hashcode.go to compute its hash value.)

You cannot change the primary IP address of an interface. It requires creating a new interface. This is an AWS restriction.

The primary private IP is returned in a dedicated field when querying the interface from AWS along with an ordered list of secondaries. Upon creation, if private_ips was unset, it is populated with the union of the primary ip and secondaries and is exported. The primary ip at that point looks just like any other address on the interface. This can be considered a feature: you really do not have to ensure you put IPs in any particular order in the config file and as long as the set is the same no "unnecessary" change will occur to reorder them.

If the addresses in private_ips are changed then terraform applies the diff to the interface by assigning or unassigning addresses. If the customer removes the primary ip from the list, the provider is not currently smart enough to realize that it cannot unassign the primary ip. When it tries to do that terraform returns an error from AWS #14366. So it is important to note that the provider does not determine if a new primary would be selected using the hash method described above.

The implication is that if you are starting from a blank slate, the provider keeps the primary consistent, but if you already have a primary and you add to the list, it updates to make sure you have the IPs you requested but it does not insist on a more disruptive change to the primary in order to be consistent with choosing the same primary ip that an equivalent deployment from scratch would.

If you remove private_ips, then the provider considers the current list on the interface to be what you want and indicates a change to the computed (output) state if the currently recorded list is different from that.

private_ips_count

private_ips_count can be used to set the number of secondary private IPs, which private_ips also includes the primary so they do not match as expected, #996.
Currently when private_ips_count changes, the private_ips are not refreshed and vice versa. That is also a bug as reported in issue #9277.
The count can be set upon creation and all IPs are chosen by AWS. If there are already private IPs configured, then altering private_ips_count will add or remove existing IPs accordingly.

If you remove private_ips_count, then the provider considers the current existing count on the resource to be what you want and indicates a change to the computed count if the currently recorded count is different from that.

Combining changes to private_ips and private_ips_count

The AWS provider currently allows both private_ips and private_ips_count to exist in the configuration file. However, this only makes sense if you are willing to change them in sequence.

There are existing workflows for increasing and decreasing the current number of IP addresses that appear to work and should be preserved. When private_ips is in use, setting a higher private_ips_count will result in assigning new IPs. If the user then immediately adds these new Ip addresses to the private_ips in the config file, then that works. Lowering the count can work in the same way. If the user does not immediately correct the configuration, however, the state will never converge.

There are also these problems:

• If you try to initially create an interface with both settings, it produces an erroneous request to AWS.
• If you had a list of private_ips and an equivalent count set at three and then add an entry to the list (4) and simultaneously decrease the count (2), you actually end up with three (they do conflicting operations and neither setting is honored).

private_ip

private_ip can be used to read the primary private ip address of the interface. It appears to be undocumented, however it is fairly essential because private_ips is not enough. It is probably used widely enough at this point that it would be disruptive to remove.

private_ip is accepted in the config file but has no effect. In this case it also no longer reflects back the current private_ip of the interface after a create or update! It just reads as the value you set - the one that was ignored. Since this variable may be present in existing configurations without ill-effect, it would be disruptive to start disallowing it.

ipv6_addresses and ipv6_address_count

These settings operate in a very similar manner as their ipv4 counterparts. However, they are not allowed in the config file simultaneously and there are no limitations on removing IPv6 addresses from an active interface. This greatly simplifies the issues to be resolved.

Customer Problems

• With multiple IPs, you cannot specify the primary ip.
• With multiple IPs, you cannot specify the order of the secondaries.
• With multiple IPs, you cannot read the order of the secondaries (although that is available via a separate data source resource).
• If the primary IP is removed from the set of private_ips then it fails with an error during execution
• private_ips is not updated if private_ips_count changes so you cannot determine what IPs you just got assigned
• private_ips_count is not updated if private_ips changes so you cannot rely on this setting for a correct count
• Initial settings including both private_ips and private_ips_count fails with an error during execution
• Simultaneous changes to private_ips and private_ips_count produces unreasonable results

Requirements

• All of the problems listed above must be resolved.
• The solution must be backwards compatible in that it does not break existing configurations.
• If there are new settings, the customer should be able to seemlesly transition from their current configuration
• Customer must be able to revert back to their existing settings
• New Acceptance Tests
• Documentation

Proposed Solution

Some have suggested that terraform should use the first address in private_ips as the primary, resolving this issue, and the rest of the private_ips should be secondary IPs that match the ordering in AWS, resolving #836 and many others.

The fact that the returned list is not ordered has been reasonably reported to be a bug and has even been triaged and labeled as such several times.

Changing the behavior of private_ips at this point is probably not going to be accepted and nor should it be, in my opinion. It is a significant breaking-change and a pull request #1749 that does so was labeled as such probably for the reasons explained in this comment #1749 (comment). Fixing it would likely result in many network interfaces being destroyed and recreated after someone upgrades and "restacks".

So this perfectly reasonable request to select a primary ip amongst a set has been open over two years because resolving it is apparently tied to fixing a bug which is, well, unfixable.

But I think there is a way forward if we start with the (probably unpopular) assumption that the current behavior of private_ips is actually not a bug, or at least one that can be fixed. It is reasonable to expect it to work differently, i.e. for the order to be respected. That the limited documentation is actually misleading does not help the matter. #10506

The proposed solution starts with accepting the current established behavior: use private_ips if you want to manage the IPs as a set. But introduce new settings that allow managing IPs as a list.

Specifically, a new setting private_ip_list is proposed for managing the private IPs as an ordered list which is tightly bound to the ordering in AWS. You can only use one method or the other. Due to backwards compatibilty requirements and other restrictions, a new boolean setting private_ip_list_enabled is also introduced that controls which settings are in control. If private_ip_list is enabled then it controls the configuration and private_ips and private_ips_count will refect the update with corresponding values. If private_ip_list is not enabled then private_ips and private_ips_count control the configuration like they do now and private_ip_list only reflects those settings.

It is worth noting that for data_source_aws_network_interface, private_ips is already an ordered list that respects what is returned from AWS. So nothing needs to change there.

Specification

The existing settings private_ips, private_ip, and private_ips_count would continue to work the same way. There are four bug fixes though:

• private_ips now updates after private_ips_count is changed
• changing private_ips only forces a new interface if the current primary ip is not in the new set of private_ips.
• If you change both the private_ips set and the private_ips_count simultaneously the IPs are set and then the count is applied correctly. This way you can add specific members and extend simultaneously. The current code calculated the wrong number of IPs to add.
• If you set both private_ips and private_ips_count for a new network interface then the same process occurs. The current code causes an AWS error.

There would be four new arguments:

• private_ip_list (list of strings)
• private_ip_list_enabled (boolean)
• ipv6_address_list (list of strings)
• ipv6_address_list_enabled (boolean)

In the config file private_ips and private_ips_count will conflict with private_ip_list and the same goes for ipv6. This is to avoid confusion and unnecessary "phantom" updates that would likely result.

When private_ip_list_enabled is true, then private_ip_list controls the ordering of the IPs and private_ips and private_ips_count are computed. If private_ip_list_enabled is false or missing then private_ips and private_ips_count are in control and private_ip_list is computed. If private_ip_list is present in the config file, then neither private_ips nor private_ips_count are allowed in the config file.

When private_ip_list_enabled is true:

• The first address of private_ip_list will be the primary.
• The rest of the IPs will be the secondaries in the same order returned from AWS.
• The network interface is replaced if and only if the first address is changed.
• When the user changes secondaries then all secondaries are unassigned and the new ones reassigned back in order, one-by-one (which appears necessary to produce the proper order in AWS).

To aid transitions between settings, both would be exported and will be kept up-to-date. Therefore, if a user decides to change the config from a set to a list, they can create an output or take the list from "terraform show" which would be in the proper order. Switching to a list which is already in the current order will not produce a resource update. Switching from a list to a set which has the same IPs present as the current list will not produce a resource update.

There is a dev and test plan in the forthcoming PR.

Issues resolved

I am commenting here because the lack of an ability to select the primary IP among multiple private IPs seems to have received the most interest.

The other issues the forthcoming PR closes are #169 and #836 and #9277 and #10506 and #14366.

Issue #996 looks resolved as far as I can tell. Finally, issue #1490 can probably be closed. It was a user configuration problem thats gone stale.

Thanks

Thanks to @nbaztec for the previous PR #1749 which was helpful to reference, especially the tests. Also, thanks to Hashicorp for investing time in their developer docs.

[github-actions]

This functionality has been released in v3.74.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.