Get Private IP of instance launched by autoscaling group

[hashibot]

This issue was originally opened by @coolgooze as hashicorp/terraform#11713. It was migrated here as part of the provider split. The original body of the issue is below.

---

ASG
resource "aws_autoscaling_group" "REDIS_ASG" {
name = "${var.environment}-REDIS_ASG"
launch_configuration = "${aws_launch_configuration.redis_launch_conf.name}"
#availability_zones = ["us-west-2a", "us-west-2b", "us-west-2c"]
vpc_zone_identifier = ["${data.aws_subnet.PrivateDBSubnetAZ1.id}","${data.aws_subnet.PrivateDBSubnetAZ2.id}","${data.aws_subnet.PrivateDBSubnetAZ3.id}"]
min_size = 1
max_size = 1
desired_capacity = 1
health_check_grace_period = 600
health_check_type = "EC2"
force_delete = "false"
termination_policies = ["OldestInstance"]
tag {
key = "Name"
value = "${var.environment}-int-redis"
propagate_at_launch = true
}
}

i am trying to create a A Record
resource "aws_route53_record" "redis" {
zone_id = "${data.aws_route53_zone.dns.zone_id}"
name = "redis-${var.environment}.${data.aws_route53_zone.dns.name}"
type = "A"
ttl = "60"
records = ["${aws_autoscaling_group.REDIS_ASG.private_ip}"]
}

Output-

• Resource 'aws_autoscaling_group.REDIS_ASG' does not have attribute 'private_ip' for variable 'aws_autoscaling_group.REDIS_ASG.private_ip'

[jasonkuehl]

I just had a need for this today. I would love for this to be a real thing.

[brandonawells]

I ran into a need for this today. I'd love this to exist. Any updates on the request?

[catsby]

Hello – Can you describe the use case here? I'm curious why you would want the direct IP of any instance that's in an autoscaling group, as opposed to connecting via a Load Balancer.

If I'm not mistaken a specific advantage of an ASG is ensuring you have a certain number of instances available and running, not that any given one of them exists at a certain IP.

If the ASG resource were to export a private IP, and on AWS side the ASG were to change instances for any reason, that IP could become stale until your next plan and apply, right? In which case there could be a window of time when your A record routes you to a bad IP, correct? Where as if the A record pointed to an IP that's assigned to a load balance, the roll over would happen automatically.

I'm curious how exposing this would be used in a reliable, stable way. Thanks!

[jasonkuehl]

@catsby You are correct in saying that IP would be stale and Terraform would need to run in order to update the A record attached to the Route53 record.

We have use case that we have a set of servers that run different jobs. Everything is controlled in puppet and Terraform. Right now we have a module that builds off a count (normally). What we wanted to do was put them into an ASG in order to have self healing in case a node would fail. However we cant move forward with testing due to there is no output of private ips or a node list.

We started an effect sometime ago to remove IP Address from configs and swap to DNS names. It would be icing on the cake if we can build a way to self heal these servers in case of aws instance failure using ASG.

[jainishshah17]

+1

[bdirito]

Another use case
https://github.com/terraform-community-modules/tf_aws_bastion_s3_keys

This sets up a bastion in an autoscaling group min=max=desired=1. The reasoning behind this is to autorelaunch the bastion if it goes down. W/o this however I have to do a check after to see what ip the bastion actually got.

[bernadinm]

It would be very useful to have this feature. Users can run types of one off workloads with a list of known IPs for the ASG within a null_resource. +1 for me as well.

[igoratencompass]

@catsby Look at my comment in the closed issue migrated here: hashicorp/terraform#11713

To repeat:

Very useful, will provide same functionality as CloudFormation does. We can simply wait for ASG to finish creating all instances and report back with private ip's, instance id's etc. so we can have further actions follow in terraform like creating dns records for the instances etc.

Also no one says that ASG has to be related to an ELB, in our case we have the instances behind HAProxy so we need to know their IP's.

This has been opened since February, is it really that difficult to resolve?

[igoratencompass]

@catsby Another example, I want to create Route53 DNS records and health checks after launching instances:

# Host-specific A records for both load-balancers
resource "aws_route53_record" "service-record" {
   zone_id = "<my-zone-id>"
   count = 2
   name = "service-lb-${format("%03d", count.index + 1)}.mydomain.com"
   type = "A"
   ttl = "60"
   records = ["${element(aws_instance.lb-service.*.public_ip, count.index)}"]
}

# Health checks for each of the load balancers
resource "aws_route53_health_check" "service-healthcheck" {
  ip_address = "${element(aws_instance.lb-service.*.public_ip, count.index)}"
  count = 2
  port = 50000
  type = "HTTP"
  resource_path = "/health"
  failure_threshold = "5"
  request_interval = "30"
  tags = {
    Name = "service-${format("%03d", count.index + 1)}.production"
  }
}

# Group consisting of 2 alias records to the host records, with associated health checks, having weighted routing
resource "aws_route53_record" "service-group" {
   zone_id = "<my-zone-id>"
   count = 2
   name = "service.mydomain.com"
   type = "A"
   weighted_routing_policy = {
      weight = "50"
   }
   health_check_id = "${element(aws_route53_health_check.service-healthcheck.*.id, count.index)}"
   set_identifier = "service${format("%03d", count.index + 1)}"
   alias {
     name = "${element(aws_route53_record.service-record.*.fqdn, count.index)}"
     zone_id = "<my-zone-id>"
     evaluate_target_health = true
   }
}

When instances are launched via ASG there is no way to do this. We are forced to move this code into user-data instead which is big PITA.

[hernan82arg]

+1

[igoratencompass]

Any update on this?

[catsby]

I'm not opposed to this feature in general but I do have opinions on how it should be implemented. I'll also be upfront and say that we (HashiCorp) are unlikely to get to this feature soon, but gladly welcome contributions if someone can pick it up.

As for how to implement:

• The DescribeAutoScalingGroups endpoint returns a list of ASGs (we query by the specific name, so we get just the one, if it exists)
• The AutoScalingGroup has a attribute that lists it's Instances
• That list contains Instance information for each instance, but it is incomplete information. Instance IP Addresses (public or private) are not included here

So the problem here in short is that the IP Addresses are not returned by default and you need to do a handful of additional API calls to get them. For large setups, these extra API calls add up quickly, so by default, we should not include Instance IP Addresses in the state of AutoScaling Groups. I recognize it's a useful feature but also think it will not be an often used one, so by default we shouldn't consume all these API calls.

That said, I believe/propose we can support this functionality with a data source:

• data.aws_autoscaling_group.group (singular) not to be confused with data.aws_autoscaling_groups (plural)
• this data source should provide more information about the specific ASG (as opposed to data.aws_autoscaling_groups which just returns names)
• should provide and optional, though default false, attribute like get_instance_properties to instruct the data source to look up instance attributes:
  • information on Instances we get from DescribeAutoScalingGroup > Instances
  • information returned from DescribeInstances

Alternatively we could expand the data.aws_autoscaling_groups data source, but I think I'd prefer the separation.

To get the IP address(es) for the Instance(s) we need to call DescribeInstances for each one. Fortunately we can use an EC2Filter and filter by requester-id, in this case, the AutoScaling Group Id, so we don't have to group up the instance-ids from the DescribeASG output and then feed that into the DescribeInstances call.

I believe this new data source would keep this instance information in a TypeSet of Instances. Unfortunately you can't directly reference the value in a set with things like element(set, count). So if you want a list of ip addresses, we may want to take the results of the DescribeInstances call and aggregate all the public ips into a calculated field public_ips , and private ips into private_ips.

So in the end the out put of this data source would be like this:

• asg info
• set of instances
• list of public ips
• list of private ips

How does this sound?

resource "autoscaling_group" "example" {
  [...]
}

data "autoscaling_group" "example_ips" {
  autoscaling_id = "${aws_autoscaling_group.example.id}"
  # Fetch Instance information
  # Retrieves instance information
  # Retrieves public_ip's
  # Retrieves private_ip's
  get_instance_properties = true
}

resource "aws_route53_record" "service-record" {
   zone_id = "<my-zone-id>"
   count = 2
   name = "service-lb-${format("%03d", count.index + 1)}.mydomain.com"
   type = "A"
   ttl = "60"
   records = ["${element(data.aws_autoscaling_group.example_ips.private_ips, count.index)}"]
}

[jasonkuehl]

@catsby I love this and it would help a lot.

[igoratencompass]

@catsby beautiful!

[samsixtyone]

any update on this, in real need of this solution.

[im2kl]

Has this been looked at? a complete stopper on what im required to do.

[alona-shevliakova]

any update on this?

[peterloron]

This would be amazing to have!

[n3ph]

This would be glorious! Since Lambda triggered by CloudWatchEvent is a mess!

[lebedevdsl]

Need this one too

[SushantJagdale]

Anyone find workaround or solution for this issue? This is completely show stopper for what I need to do.

[JeremieBethmont]

+1

[xvillanu]

+1

[uriinf]

+1

[drnorton83]

+1

[cleanshavenalex]

+1

[Alifener]

+1

[tomaszwostal]

+1

[sadok-f]

+1

[tlipatov]

Here are some solutions that I implement:

1. Use the module terraform-aws-modules/autoscaling/aws to launch my ASG's
   Tag the ASG and then use the aws_instance data source to search for the instances based on the tags. Then do what ever I need to do with the data

2. When using an ASG for a bastion host. I create an iam_instance_profile for the ASG that allows the instance itself to update its own route53 record when it boots up using a launch config script.

Hope this helps

[JoaoManoel]

+1

[ricardo-larosa]

+1

[lexsys27]

Is there currently any workaround available?

[lexsys27]

This way allowed me to get ips of worker nodes:

data "aws_instances" "workers" {
  instance_tags {
    Name = "lexsys-eks-asg"
  }
}

output "private-ips" {
  value = "${data.aws_instances.workers.private_ips}"
}

output "public-ips" {
  value = "${data.aws_instances.workers.public_ips}"
}

Outputs:

private-ips = [
    10.0.0.75,
    10.0.1.87
]
public-ips = [
    52.44.215.211,
    54.153.70.110
]

Sure, this workaround has the limitations, but worked for me.

[ghost]

@catsby One use case is for cluster setup. In such a scenario we want to be able to only provide the SSH keys of the instances to people who want to test a cluster like kafka, zookeeper, elasticsearch etc without having to give them access to the AWS console. If terraform could simply output the private ip address then the troubleshooters would have a list of IP addresses to log into simply with their SSH keys and without requiring access to AWS Console.

[nunofernandes]

+1

[moosahmed]

@catsby Yeah another use case is when making a kubernetes cluster through an asg. you need the master's private ip to feed into the workers. Is there someone who knows a work around for this?

[matt9949]

+1

[npockrus-bnet]

+1

[jeandreh]

+1. My use case involves Hashicorp Vault, which needs to be initialised before being recognised as a healthy node by the load balancer.

[kamusin]

+1

[KIVagant]

Any workarounds for Google Cloud Platform? It has the same problem with google_compute_instance_group_manager.

[igoratencompass]

@lexsys27 the question is specific for when instances are created via aws_autoscaling_group

@KIVagant I'm afraid you'll need to wait patiently as the rest of us have (since Feb 2017) for this to be included in terraform, my prognoses would be around version 2.27 ... probably :-)

[lexsys27]

@igoratencompass I do use aws_autoscaling_group to create instances.

I tag the instances in the template and use this tag to retrieve IPs in the data section.

[igoratencompass]

I see was not aware of that feature of data module.

[RuBiCK]

#511 (comment) is a workaround but be carefull doc page

Note: It's strongly discouraged to use this data source for querying ephemeral instances (e.g. managed via autoscaling group), as the output may change at any time and you'd need to re-run apply every time an instance comes up or dies.

This note is other reason to implement this new data source

[syncroswitch]

you can pull all instance data with a round about lookup.

( example assuming some asg named aws_autoscaling_group.one is already defined )

data "aws_instances" "nodes" {
  depends_on = [ "aws_autoscaling_group.one" ]
  
  instance_tags {
    Name = "some_unique_tag_in_your_asg"
  }
}

data "aws_instance" "asg-one-instances" {
  count = "${ var.asg_one_count }"
  depends_on = ["data.aws_instances.nodes"]
  instance_id = "${data.aws_instances.nodes.ids[count.index]}"
}

output "private-ips" {
  value = "${ data.aws_instance.asg-one-instances.*.private_ip }"
}

output "public-ips" {
  value = "${ data.aws_instance.asg-one-instances.*.public_ip }"
}

output "private-dnsnames" {
  value = "${ data.aws_instance.asg-one-instances.*.private_dns }"
}

output "public-dnsnames" {
  value = "${ data.aws_instance.asg-one-instances.*.public_dns }"
}

but I would much prefer to have the instance data directly available in the asg attributes.

[igoratencompass]

@syncroswitch the problem I see with this is the asg_one_count variable. It is a static value which will/may not match the actual state of the ASG on consecutive runs, i.e. teh ASG has grown or shrunk in the mean time.

[claywd]

@syncroswitch @igoratencompass it works to get the endpoints and private ips when you launch, which is what is mostly needed.

Also, just set count = "${ var.asg_one_count }" to count = ${aws_autoscaling_group.one.desired_capacity}

If people truly have need of dynamic ips then they should write it up in python, bash, go, etc and call it from terraform then use that instead of the desired capacity but I doubt it will really be necessary or desired in the long run.

[minhajuddin]

I started off with @syncroswitch's config and ended with a simpler workaround:

data "aws_instances" "ecs_instances_meta" {
  instance_tags = {
    # Use whatever name you have given to your instances
    Name = var.ecs_cluster_name
  }
}

output "ecs-private-ips" {
  value = data.aws_instances.ecs_instances_meta.private_ips
}

[Xtigyro]

5 yrs later - and still unresolved?

[jasonkuehl]

A big no.

[BarbaraStanley]

Deep sigh, needed this.
data "aws_instances" "test" {
instance_tags = {
Role = "HardWorker"
}

filter {
name = "instance.group-id"
values = ["sg-12345678"]
}

instance_state_names = ["running", "stopped"]
}
Is there a way to use the filter to narrow things down, I gave the instances a prefix from the launch template