Terraform fails when adding multiple secondary private ip addresses

[arthar360]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.13.5

• provider registry.terraform.io/-/aws v3.18.0
• provider registry.terraform.io/hashicorp/aws v3.18.0

Terraform Configuration Files

# main.tf

provider "aws" {
    region = "eu-central-1"
}

resource "aws_network_interface" "test" {
  count          = 2
  subnet_id       = var.aws_cmp_subnet_id[count.index]
  security_groups = [var.aws_cmp_vpc_security_group_ids]
  private_ips_count =  2
}

module "sql01" {
    source = "git::ssh://git@bitbucket.org/xyz/terraform-aws-ec2.git"
    aws_instance_count          = 2
    aws_region                  = "eu-central-1"
    aws_ami                     = "ami-0da198ce94f235cdc"
    aws_instance_type           = "t2.large"
    aws_network_interface    = element(aws_network_interface.test[*].id, 0)

}

module "sql02" {
    source = "git::ssh://git@bitbucket.org/xyz/terraform-aws-ec2.git"
    aws_instance_count          = 2
    aws_region                  = "eu-central-1"
    aws_ami                     = "ami-0da198ce94f235cdc"
    aws_instance_type           = "t2.large"
    aws_network_interface     = element(aws_network_interface.test[*].id, 1)
}


# Module source file at source = "git::ssh://git@bitbucket.org/xyz/terraform-aws-ec2.git"

provider "aws" {
    region = var.aws_region
}

resource "aws_instance" "server" {
    count                       = var.aws_instance_count
    ami                         = var.aws_ami
    secondary_private_ips       = var.aws_secondary_private_ips
    instance_type               = var.aws_instance_type
    network_interface {
      network_interface_id = var.aws_network_interface
      device_index         = 0
    }
}

Instead of aws_network_interface = element(aws_network_interface.test[*].id, 1) I have also tried aws_network_interface = aws_network_interface.test[1].id. It returns the same error.

Debug Output

https://gist.github.com/arthar360/02f25e4a1f272ae443a29ee1aa525d00

Panic Output

crash.log

Expected Behavior

The 2 ec2 instances should have been created with 3 private ip addresses each.

Actual Behavior

Terraform created successfully and failed with error

module.sql01.aws_instance.server[1]: Still creating... [20s elapsed]
module.sql02.aws_instance.server[0]: Still creating... [20s elapsed]
module.sql01.aws_instance.server[1]: Creation complete after 23s [id=i-062af6a475130f9bf]
module.sql02.aws_instance.server[0]: Still creating... [30s elapsed]
module.sql02.aws_instance.server[0]: Creation complete after 33s [id=i-0b0351f427c10c57e]

Error: Error waiting for instance (i-09baa706e3364263e) to become ready: Failed to reach target state. Reason: Server.InternalError: Internal error on launch



Error: Error waiting for instance (i-0b72b8b6e25a9c41a) to become ready: Failed to reach target state. Reason: Server.InternalError: Internal error on launch

On subsequent terraform apply, Terraform crashes. Please refer to debug gist for that

It is also strange to notice that instance id's shown in
Creation complete after 23s [id=i-062af6a475130f9bf] and Creation complete after 33s [id=i-0b0351f427c10c57e]
are different than
Error waiting for instance (i-09baa706e3364263e) to become ready and Error waiting for instance (i-0b72b8b6e25a9c41a) to become ready

References

• A similar issue is listed at Terraform Instance creation for ENI with Secondary private private IPs greater than 1 Fails #1490 Open more than 3 years ago and has no resolution.

[ewbankkit]

panic: interface conversion: interface {} is nil, not []map[string]interface {}
2020-11-27T21:33:47.541+0100 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: 
2020-11-27T21:33:47.541+0100 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: goroutine 73 [running]:
2020-11-27T21:33:47.541+0100 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.readBlockDevices(0xc001322200, 0xc0003fcc00, 0xc001a16898, 0x0, 0x0)
2020-11-27T21:33:47.541+0100 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-aws/aws/resource_aws_instance.go:1599 +0x751
2020-11-27T21:33:47.541+0100 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsInstanceRead(0xc001322200, 0x61b9460, 0xc001cfd080, 0xb1f7760, 0x6f66e48)
2020-11-27T21:33:47.541+0100 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5:  /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-aws/aws/resource_aws_instance.go:903 +0x166c
2020-11-27T21:33:47.542+0100 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0xc00079d8c0, 0x7a00c60, 0xc001e9ee80, 0xc001322200, 0x61b9460, 0xc001cfd080, 0x0, 0x0, 0x0)
2020-11-27T21:33:47.542+0100 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5:  /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.3.0/helper/schema/resource.go:280 +0x88
2020-11-27T21:33:47.542+0100 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc00079d8c0, 0x7a00c60, 0xc001e9ee80, 0xc00196aee0, 0x61b9460, 0xc001cfd080, 0xc000309248, 0x0, 0x0, 0x0)
2020-11-27T21:33:47.542+0100 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5:  /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.3.0/helper/schema/resource.go:554 +0x1c2
2020-11-27T21:33:47.542+0100 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0xc000ecfdc0, 0x7a00c60, 0xc001e9ee80, 0xc001e9eec0, 0xc001e9ee80, 0x69c19e0, 0x6c48d20)
2020-11-27T21:33:47.542+0100 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5:  /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.3.0/helper/schema/grpc_provider.go:571 +0x42f
2020-11-27T21:33:47.542+0100 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).ReadResource(0xc0017b1600, 0x7a00c60, 0xc001e9ee80, 0xc001ea0840, 0xc0017b1600, 0x0, 0xc001072ba0)
2020-11-27T21:33:47.542+0100 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5:  /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.1.0/tfprotov5/server/server.go:297 +0x101
2020-11-27T21:33:47.542+0100 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler(0x6c48d20, 0xc0017b1600, 0x7a00d20, 0xc00189e870, 0xc001ea07e0, 0x0, 0x7a00d20, 0xc00189e870, 0xc001508b00, 0x534)
2020-11-27T21:33:47.542+0100 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5:  /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.1.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:344 +0x217
2020-11-27T21:33:47.542+0100 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: google.golang.org/grpc.(*Server).processUnaryRPC(0xc000373dc0, 0x7a25740, 0xc000f0f200, 0xc0019dd100, 0xc001089a40, 0xb1bbef0, 0x0, 0x0, 0x0)
2020-11-27T21:33:47.542+0100 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5:  /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1194 +0x50a
2020-11-27T21:33:47.542+0100 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: google.golang.org/grpc.(*Server).handleStream(0xc000373dc0, 0x7a25740, 0xc000f0f200, 0xc0019dd100, 0x0)
2020-11-27T21:33:47.542+0100 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5:  /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1517 +0xcfd
2020-11-27T21:33:47.542+0100 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0007b01f0, 0xc000373dc0, 0x7a25740, 0xc000f0f200, 0xc0019dd100)
2020-11-27T21:33:47.542+0100 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5:  /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:859 +0xa1
2020-11-27T21:33:47.542+0100 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5: created by google.golang.org/grpc.(*Server).serveStreams.func1
2020-11-27T21:33:47.542+0100 [DEBUG] plugin.terraform-provider-aws_v3.18.0_x5:  /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:857 +0x204

terraform-provider-aws/aws/resource_aws_instance.go

Lines 1598 to 1600 in 79e4383

	if _, ok := d.GetOk("ebs_block_device"); ok {
	if len(ibds["ebs"].([]map[string]interface{})) == 0 {
	ebs := make(map[string]interface{})

ibds is nil from here

terraform-provider-aws/aws/resource_aws_instance.go

Lines 1679 to 1681 in 79e4383

	if len(instanceBlockDevices) == 0 {
	return nil, nil
	}

[ewbankkit]

@arthar360 Thanks for raising this issue.
It looks like the calls to sql01 and sql02 are each requesting 2 instances (aws_instance_count = 2) but specify only a single ENI ID (aws_network_interface = element(aws_network_interface.test[*].id, n)), so 2 instances are created with the same ENI and the second fails with the Server.InternalError: Internal error on launch error (hence the total of 4 instance IDs reported, 2 of which result in failed instances).
The crash on refresh should not happen; that's a bug.

[arthar360]

Thanks @ewbankkit , setting aws_instance_count = 1 for both sql01 and sql02 solved it.

[github-actions]

This functionality has been released in v3.74.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.