Error updating Autoscaling group: InvalidQueryParameter: Invalid launch template: When a network interface is provided, the security groups must be a part of it.

[djdevin]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.7

• provider.aws v1.19.0

Affected Resource(s)

aws_autoscaling_group
aws_launch_template

Terraform Configuration Files

# Launch template for nodes.
resource "aws_launch_template" "nodes" {
  name = "nodes"

  block_device_mappings {
    device_name = "/dev/sda1"

    ebs {
      volume_size = 32
      volume_type = "gp2"
    }
  }

  credit_specification {
    cpu_credits = "unlimited"
  }

  disable_api_termination = true

  ebs_optimized = true

  iam_instance_profile {
    name = "${aws_iam_instance_profile.nodes.id}"
  }

  # Currently Fedora 28
  image_id = "ami-e754e298"

  instance_initiated_shutdown_behavior = "terminate"

  instance_type = "${terraform.env == "prod" ? "t2.xlarge" : "t2.large"}"

  key_name = "ansible"

  monitoring {
    enabled = true
  }

  network_interfaces {
    device_index                = 0
    associate_public_ip_address = true
    security_groups             = ["${aws_security_group.openshift.id}"]
  }

  placement {
    availability_zone = "us-east-1a"
  }

  vpc_security_group_ids = ["${aws_security_group.openshift.id}"]

  tag_specifications {
    tags {
      Name = "node"
    }
  }
}

resource "aws_autoscaling_group" "nodes" {
  name = "OpenShift Nodes"

  launch_template = {
    id = "${aws_launch_template.nodes.id}"

    version = "$$Latest"
  }

  min_size           = "${terraform.env == "prod" ? "3" : "3"}"
  max_size           = "${terraform.env == "prod" ? "3" : "3"}"
  force_delete       = 1
  availability_zones = ["us-east-1a"]

  tag {
    key                 = "Name"
    value               = "node"
    propagate_at_launch = true
  }
}

Debug Output

* aws_autoscaling_group.nodes: 1 error(s) occurred:

* aws_autoscaling_group.nodes: Error updating Autoscaling group: InvalidQueryParameter: Invalid launch template: When a network interface is provided, the security groups must be a part of it.
        status code: 400, request id: 7a98dd3b-59f8-11e8-b39f-cdb9cee0f1ee

Important Factoids

It seems like the SGs are getting into the template, but I'm not sure why the ASG is complaining about the LT.

This is updating an ASG that previously had an LC.

References

#4364

[djdevin]

I get the same error in the EC2 console trying to create an ASG from the LT. I've put the security group everywhere I can think of. What am I missing?

[djdevin]

I reproduced this without terraform so I think this is an AWS issue, but this is interesting:

1. Create a launch template and specify security groups in both the network interface and also the "security groups" section at the bottom of the template
2. Create an ASG from this launch template
3. Boom, error.

Now, go back and edit the launch template.

1. Remove the SG from the bottom, not the interface.
2. Save the LT and go back and create a new ASG from the new revision.
3. Works, ASG created.

So I removed vpc_security_group_ids and now I get
#4553

[julian-alarcon]

Hi! Is there any workaround?

I commented the vpc_security_group_ids on the launch template configuration, and now it's working the deployment of the launch template and the auto scaling group, but I don't know if this is a big change on behavior.

[joaoclaudioone]

Greetings,

Is there another workaround? Comment vpc_security_group_ids, did not work for me.

[Yashiroo]

Also having the same issue here. I don't think we should get rid of security groups, but even so, it is weird that it's complaining about SGs when they are mentioned in both network_interfaces and vpc_security_group_ids.
Anyone has more feedback or workarounds regarding this issue?

[Yashiroo]

Update:
Ok this is how i solved this for me. I kept the vpc_security_group_ids in place and removed security_groups from network_interfaces. Then updated the ASG configuration:

launch_template = {
    id         = "${aws_launch_template.nodes.id}"
    version = "${aws_launch_template.nodes.latest_version}"
  }

The issue for me was that this was using the default version of the Launch Template instead of the latest one (as version increases automatically in aws even if you don't specify it).

[afalko]

I tried doing what @Yashiroo did, but I still get there error. Did you clear terraform and manually delete the resources?

This worked for me, but only after I attempted Yashiroo's workaround:

...
  #vpc_security_group_ids = ["${aws_security_group.jenkins-nodes.id}"]
  network_interfaces {
    associate_public_ip_address = true
    security_groups = ["${aws_security_group.jenkins-nodes.id}"]
  }
...

[Yashiroo]

@afalko Sorry, i think my comment was incomplete.
Yes i deleted the environment and recreated but using terraform, no manual operations. But after doing it again today, i still get the error, so either specifying SGs or removing network_interfaces works. I can afford removing it that since i only wanted to disable public ip for the instance, which obviously is disabled by default for instances not within default VPC.
Anyhow, I think i missed this from aws documentation:

The following are limitations when creating a launch template for use with an Auto Scaling group:
You cannot specify multiple network interfaces.
If you specify a network interface, its device index must be 0.
If you specify a network interface, you must specify any security groups as part of the network interface, and not in the Security Groups section of the template.
You cannot specify private IP addresses.
You cannot use host placement affinity.
If you specify Spot Instances, you must specify a one-time request with no end date.

The problem is, this is misleading since in terraform documentation, the example shows the usage of network_interfaces this way:

network_interfaces {
    associate_public_ip_address = true
  }

But this does not work (at least did not work for me) and terraform complains about security groups that need to be added, even though no network interface was mentioned in that block.
I hope someone takes a look at this and provides some insight for us.

[shapeofarchitect]

This is indeed the same in my configuration , which is exactly the same as what @Yashiroo mentioned. I have tried multiple approaches.
Below are some I tried.

• only added security_groups under network_interfaces and it fails with the error.
• Tried only having Only vpc_security_group_ids with security groups ids in list form and it also gives the same error.

So I guess it's not fixed yet. I would check net on AWS console to test AWS Launch Template and ASG binding but I suspect it's the AWS API that's causing the failure. I will update after my tests.

[lds1804]

Update:
Ok this is how i solved this for me. I kept the vpc_security_group_ids in place and removed security_groups from network_interfaces. Then updated the ASG configuration:

launch_template = {
    id         = "${aws_launch_template.nodes.id}"
    version = "${aws_launch_template.nodes.latest_version}"
  }

The issue for me was that this was using the default version of the Launch Template instead of the latest one (as version increases automatically in aws even if you don't specify it).
I removed all the network interface section and worked for me. I only needed a private ip for my machines. Thanks for sharing your solution.

[honzous]

I was getting this error with this line within the network_interfaces section:

ipv4_address_count = 1

When I removed this line, ASG deployed.

[edaemon]

I was also able to get around this by removing vpc_security_group_ids from the aws_launch_template and adding them instead in the network_interfaces block. I also had to include the subnet in the vpc_zone_identifier list in the aws_autoscaling_group and I used the ${aws_launch_template.nodes.latest_version} format.

To provide what seems to be a working example of the necessary parts:

resource "aws_launch_template" "example" {
  ...
  # Do not include vpc_security_group_ids
  network_interfaces {
    associate_public_ip_address = true
    security_groups             = ["${aws_security_group.example.id}"]
    subnet_id                   = "${aws_subnet.example.id}"
  }
}

resource "aws_autoscaling_group" "example" {
  ...
  vpc_zone_identifier = ["${aws_subnet.example.id}"]

  launch_template = {
    id      = "${aws_launch_template.example.id}"
    version = "${aws_launch_template.example.latest_version}"
  }
}

I think this comes from a limitation in the EC2 API where instance security groups aren't compatible with network interfaces with public IP addresses (only the network interface will have a security group), but the vague error message makes me unsure.

[ghost]

I encountered same problem and commenting out vpc_security_group_ids worked for me:

resource "aws_launch_template" "lt" {
  ...
  network_interfaces {
    ...
    security_groups = ["${aws_security_group.instance.id}"]
  }
  # vpc_security_group_ids = ["${aws_security_group.instance.id}"]
  ...
}
resource "aws_autoscaling_group" "asg" {
  ...
  launch_template {
    id = "${aws_launch_template.lt.id}"
    version = "$$Latest"
  }
  ...
}

[ghost]

I used another approach - switch back to EC2 Classic mode in the ASG - i.e. this is what I did:

1. attach a security group to the ENI on creation
2. remove all security group references in autoscaling_group, launch_template and network_interface section of launch_template
3. remove the vpc_zone_identifier from autoscaling_group
4. add availability_zones to autoscaling_group

Works perfectly! Only challenge for me is now to bring the list with availability_zones for autoscaling_group and the list of network_interfaces for launch_template into the same order - otherwise AWS will complain about az not matching between ENI and ASG...

[skyuuka]

Has this issue been resolved? I am having the same issue when using launch_template in Batch. I think the two issues are connected.

[julian-alarcon]

Can you please provider Terraform version and AWS Provider version @skyuuka ?

[skyuuka]

Can you please provider Terraform version and AWS Provider version @skyuuka ?

Terraform version: v0.12.20
AWS Provider version: 2.48

[julian-alarcon]

Can you please also paste the output of the error @skyuuka ?

[sporokh]

That's not a bug/issue, but intended behavior by AWS:
https://docs.aws.amazon.com/autoscaling/ec2/userguide/create-launch-template.html

If you specify a network interface, you must configure the security group as part of the network interface, and not in the Security Groups section of the template.

And confirmed by AWS (from my support ticket):

you have to explicitly set SG on the interface level that because an instance could have multiple interfaces each associated with separate security groups, otherwise if you don't specify network interfaces the instance will just get its default interface and the SGs defined in the top-level "security groups" section will just be attached as the default behavior

So in case setting network interface you just need to provide SG on the interface level and remove vpc_security_group_ids from the resource

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.