Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Instance Profile has no associated IAM Roles for some instances #2660

Closed
dpaq opened this issue Jul 8, 2015 · 22 comments · Fixed by #7938
Closed

Instance Profile has no associated IAM Roles for some instances #2660

dpaq opened this issue Jul 8, 2015 · 22 comments · Fixed by #7938

Comments

@dpaq
Copy link

dpaq commented Jul 8, 2015

I think I'm seeing a race condition when using instance profiles.

In this case there were 5 instances (foo.0, foo.1, bar.0, bar.1, bar.2) that all use the same InstanceProfile. Two of them work fine but three of them claim the instance profile has no role associated:

 aws_route_table_association.testing: Creation complete
 aws_instance.bar.1: Error: 1 error(s) occurred:

 * Error launching source instance: InvalidParameterValue: IAM Instance Profile "arn:aws:iam::123123:instance-profile/test_bucket_access_instance_profile" has no associated IAM Roles
  status code: 400, request id: []
 aws_instance.bar.2: Error: 1 error(s) occurred:

 * Error launching source instance: InvalidParameterValue: IAM Instance Profile "arn:aws:iam::123123:instance-profile/test_bucket_access_instance_profile" has no associated IAM Roles
  status code: 400, request id: []
 aws_instance.foo.0: Error: 1 error(s) occurred:

 * Error launching source instance: InvalidParameterValue: IAM Instance Profile "arn:aws:iam::123123:instance-profile/test_bucket_access_instance_profile" has no associated IAM Roles
  status code: 400, request id: []
 aws_instance.bar.0: Provisioning with 'file'...
 aws_instance.foo.1: Provisioning with 'file'...

I'm wondering if this is an eventual consistency issue and this particular error launching should just include some retry logic. Obviously the InstanceProfile does have a Role since it works for some of the instances.

This is not repeatable regularly and just applying again gets me my missing instances.

A slightly redacted portion of the config:

{
resource "aws_instance" "foo" {
  ami = "${var.baseAMI}"
  instance_type = "${var.fooInstanceType}"
  count = "${var.numFooInstances}"
  key_name = "${var.keyName}"
  subnet_id = "${aws_subnet.foo.id}"
  vpc_security_group_ids = ["${aws_security_group.default.id}", "${aws_security_group.ssh.id}"]
  tags {
      Name = "${format("foo-%d", count.index)}"
  }
  iam_instance_profile = "${aws_iam_instance_profile.test_bucket_access_instance_profile.name}"
  root_block_device {
    delete_on_termination=true
  }
  provisioner "file" {
    connection {
      key_file = "${var.sshKeyFilename}"
      user = "${var.amiUsername}"
    }
    source = "mystuff"
    destination = "/home/${var.amiUsername}/mystuff"
  }
}

resource "aws_iam_instance_profile" "test_bucket_access_instance_profile" {
    name = "test_bucket_access_instance_profile"
    roles = ["${aws_iam_role.test_bucket_access_role.name}"]
}

resource "aws_iam_role" "test_bucket_access_role" {
  name = "test_bucket_access_role"
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}

resource "aws_iam_role_policy" "test_bucket_access_policy" {
  name = "test_bucket_access_policy"
  role = "${aws_iam_role.test_bucket_access_role.id}"
  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::${var.s3BucketName}"
    },
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::${var.s3BucketName}/*"
    }
  ]
}
EOF
}
@dpaq
Copy link
Author

dpaq commented Jul 22, 2015

This was seen with 0.5.3.

@josh-padnick
Copy link

We're also seeing this issue, however we can deterministically reproduce it.

  1. Launch our AWS resources via Terraform. Works fine.
  2. Update the Launch Configuration for 3 different instances and terraform apply. This causes the following error message:
Error applying plan:

4 error(s) occurred:

* aws_instance.nat: Error launching source instance: InvalidParameterValue: IAM Instance Profile "arn:aws:iam::941285601314:instance-profile/bryan-stephens-iam-nat-profile" has no associated IAM Roles
    status code: 400, request id: []
* aws_route_table.private-app-1: InvalidInstanceID.Malformed: Invalid id: "${element(split(",", module.nat.nat_instances_string), 1 % var.num_nat_instances)}"
    status code: 400, request id: []
* aws_route_table.private-app-0: InvalidInstanceID.Malformed: Invalid id: "${element(split(",", module.nat.nat_instances_string), 0 % var.num_nat_instances)}"
    status code: 400, request id: []
* aws_route_table.private-app-2: InvalidInstanceID.Malformed: Invalid id: "${element(split(",", module.nat.nat_instances_string), 2 % var.num_nat_instances)}"
    status code: 400, request id: []

In our case, the change to Launch Configurations that we made had nothing to do with the NAT instance IAM Instance Profile mentioned in the error.

Note that we are using at least 5 different nested modules. We're on terraform 0.6.3.

Correction: This is in fact intermittent, corroborating the eventual consistency issue that @dpaq suggested. Also, we were using 0.6.2. If this issue is resolved with 0.6.3 we'll report back as such.

@gdeconto
Copy link

gdeconto commented Sep 9, 2015

we are also seeing this issue (as described by dpaq) when we deploy new AWS resources via terraform 0.6.3

@quater
Copy link

quater commented Oct 7, 2015

Encountering this issue with 0.6.3 as well. It appears to be an intermittent race condition. This problem is observed with an Terraform environment when created from scratch. Running the same terraform apply then results in a successful deployment.

@vancluever
Copy link
Contributor

+1 on this, seeing it on S3 resources possibly, upon creation of the IAM resources needed for the principal the bucket creation is performed, and when the policy application is attempted it barfs.

@catsby
Copy link
Contributor

catsby commented Nov 30, 2015

@vancluever do you have a configuration that reproduces this? If so, can you please open another issue? As it stands, IAM propagation issues need to be handled in a resource-by-resource manner, for the time being.

This particular issue (#2660) has been resolved with #2037, so I'm going to close it.

Let me know if anyone is still hitting this

@catsby catsby closed this as completed Nov 30, 2015
@vancluever
Copy link
Contributor

@catsby check out my PR #4103 for the fixes I did for this - I will re-run this with the current upstream and paste the errors, with an sanitized config (I'll just append this stuff to the PR, if that's okay).

@errordeveloper
Copy link

I am seeing this with 0.6.10.

@ickymettle
Copy link

also still seeing behaviour occasionally in 0.6.12

@bmurtagh
Copy link

+1 with @ickymettle... Seeing it occasionally in 0.6.12.

@liquid-sky
Copy link

Encountered this in 0.6.14 on every run. Had to switch back to 0.6.8 for modules to work properly.

@pdecat
Copy link
Contributor

pdecat commented May 12, 2016

Just encountered this issue with 0.6.16.

Retrying an apply worked it around.

@brikis98
Copy link
Contributor

Seeing this in 0.6.16. When combined with #5200, it makes the aws_instance resource nearly unusable if you want to attach an aws_iam_role and aws_iam_instance_profile. First "I get a diffs didn't match" bug, then after reapplying I get a "has no associated IAM Roles" bug.

@brikis98
Copy link
Contributor

@catsby Should this issue be reopened?

@tdeheurles
Copy link

@catsby
Hi, having the same issue Instance Profile has no associated IAM Roles.
I confirm that it's intermittent but raise really often. I need some terraform destroy/terraform apply to have it work.

The problem is that intermittent doesn't sound really pro :-/
Maybe it could be reopen.

@toddmichael
Copy link

Ran into this issue yesterday while creating a 4 node application stack. 1 out of 4 failed because of this issue. When the other nodes finished building, I re-ran terraform apply with no config changes and everything worked as expected. I tried again this morning with a single node, one that worked fine last night, and ran into it again. Same behavior: failed on first apply and then worked 30 seconds later when I re-ran. Definitely something awry here. If I discern a pattern, will follow up with more details. Thanks.

@tdeheurles
Copy link

Just to add an understanding of the issue: #4306.
It seems that AWS IAM are eventually consistent, so terraform have a feeback of IAM created and then we have an issue at the instance creation time.

@cmlad
Copy link

cmlad commented Jun 13, 2016

We are also seeing this on 0.6.16, for now our workaround is to add:

    provisioner "local-exec" {
      command = "sleep 90"
    }

@cmlad
Copy link

cmlad commented Jun 13, 2016

@catsby you are incorrect, #2037 fixes a slightly different problem. Here the issue is that IAM profile has propagated, but the role has not, and the error is:

* aws_instance.central: Error launching source instance: InvalidParameterValue: IAM Instance Profile "arn:aws:iam::198832413611:instance-profile/cmlad-sleep-general" has no associated IAM Roles
        status code: 400, request id: bb9f4992-4123-4a0a-8abe-f12560d71819

#2037 fixes the case where the IAM profile has not propagated. The error there is:

 aws_instance.node_instance.0: Error launching source instance: InvalidParameterValue: Value (test-cloudera-hadoop-role) for parameter iamInstanceProfile.name is invalid. Invalid IAM Instance Profile name
    status code: 400, request id:

The below retry line searches specifically for the Invalid IAM Instance Profile string which does not correspond to the error message in this issue.

if awsErr.Code() == "InvalidParameterValue" && strings.Contains(awsErr.Message(), "Invalid IAM Instance Profile") {

I will try to create a PR in the next few days. Hope you agree to reopen.

@sidhurana
Copy link

HI All.. faced same issue.. while creating e2 instance it was going through iam_instance_profile and was giving me different errors such as :- iam role already present, Invalid IAM Instance Profile name..

In my case the problem was instance type.. I was trying to launch m4.large instance in eu-west-1a and that availability zone was running full with m4.large instance types i.e you cannot launch more m4.large instances in same AZ for may be particular time period. As soon as i changed instance type, then error was gone and instance created..

IAM Instance Role creation is prior and parallel process while instance creation hence giving me strange/irrelevant IAM creation erros.

@realflash
Copy link
Contributor

@ghost
Copy link

ghost commented Apr 7, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@ghost ghost locked and limited conversation to collaborators Apr 7, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.