Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws_launch_template - panic: interface conversion: interface {} is nil, not map[string]interface {} #7885

Closed
JPScutt opened this issue Mar 11, 2019 · 8 comments · Fixed by #8844
Assignees
Labels
bug Addresses a defect in current functionality. crash Results from or addresses a Terraform crash or kernel panic. service/ec2 Issues and PRs that pertain to the ec2 service.
Milestone

Comments

@JPScutt
Copy link

JPScutt commented Mar 11, 2019

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.11
provider.aws v2.1.0

Affected Resource(s)

aws_launch_template
aws_security_group

Terraform Configuration Files

resource "aws_launch_template" "spot_instance_template" {
    count = "${var.use_spot_instance ? 1 : 0}"

    name_prefix = "${var.instance_name}-"

    iam_instance_profile {
        arn = "${var.ec2_instance_profile_arn}"
    }
    image_id = "${var.image_id}" 

    instance_market_options {
        market_type = "spot"

        spot_options {
            max_price = "${var.spot_price}"
        }        
    }
    instance_type = "${var.instance_type}"
    key_name = ""
    monitoring {
        enabled = true
    }    
    vpc_security_group_ids = [
        "${var.sg_ids}"
    ]

    tag_specifications {
        resource_type = "instance"
        tags = "${merge(var.default_tags, map("Name", "${var.instance_name}"))}"
    }

    tag_specifications {
        resource_type = "volume"
        tags = "${merge(var.default_tags, map("Name", "${var.instance_name}"))}"
    }

    tags        = "${var.default_tags}"
    user_data   = "${base64encode(var.user_data)}"

    lifecycle {
        create_before_destroy = true

        ignore_changes = [        
          "tags"
        ]
    }
}

resource "aws_launch_template" "ondemand_instance_template" {
    count = "${var.use_spot_instance ? 0 : 1}"

    name_prefix = "${var.instance_name}-"

    iam_instance_profile {
        arn = "${var.ec2_instance_profile_arn}"
    }
    
    image_id        = "${var.image_id}"    
    instance_type   = "${var.instance_type}"
    key_name        = ""

    monitoring {
        enabled = true
    }

    vpc_security_group_ids = [
        "${var.sg_ids}"
    ]

    tag_specifications {
        resource_type   = "instance"
        tags            = "${merge(var.default_tags, map("Name", "${var.instance_name}"))}"
    }

    tags        = "${var.default_tags}"
    user_data   = "${base64encode(var.user_data)}"

    lifecycle {
        create_before_destroy = true

        ignore_changes = [        
          "tags"
        ]
    }
}

resource "aws_autoscaling_group" "spot_instance_asg" {    
    count                       = "${var.use_spot_instance ? 1 : 0}"
    name                        = "${aws_launch_template.spot_instance_template.name}-asg"    
    vpc_zone_identifier         = ["${var.vpc_zone_identifier}"]
    max_size                    = "${var.max_size}"
    min_size                    = "${var.min_size}"    
    target_group_arns           = ["${var.target_group_arns}"]

    launch_template {
        id      = "${aws_launch_template.spot_instance_template.id}"
        version = "$$Latest"
    }
    
    tags = ["${concat(list(map("key", "Name", "value", "${var.instance_name}", "propagate_at_launch", true)), local.asg_tags)}"]
    

    lifecycle {
        create_before_destroy = true
        ignore_changes = [        
          "tags"
        ]
    }
}

resource "aws_autoscaling_group" "ondemand_instance_asg" {
    count                       = "${var.use_spot_instance ? 0 : 1}"
    name                        = "${aws_launch_template.ondemand_instance_template.name}-asg"    
    vpc_zone_identifier         = ["${var.vpc_zone_identifier}"]
    max_size                    = "${var.max_size}"
    min_size                    = "${var.min_size}"    
    target_group_arns           = ["${var.target_group_arns}"]

    launch_template {
        id      = "${aws_launch_template.ondemand_instance_template.id}"
        version = "$$Latest"
    }
    
    tags = ["${concat(list(map("key", "Name", "value", "${var.instance_name}", "propagate_at_launch", true)), local.asg_tags)}"]

    lifecycle {
        create_before_destroy = true

        ignore_changes = [        
          "tags"
        ]
    }
}

resource "aws_autoscaling_policy" "spot_instance_as_policy" {
    depends_on = [
        "aws_autoscaling_group.spot_instance_asg",
        "aws_launch_template.spot_instance_template"
    ]
    count                   = "${var.use_spot_instance && var.use_auto_scaling_policy ? 1 : 0}"
    name                    = "${var.application}-${var.environment}-${var.instance_name}-as-policy"  
    policy_type             = "TargetTrackingScaling"    
    autoscaling_group_name  = "${aws_autoscaling_group.spot_instance_asg.name}"

    target_tracking_configuration {
        predefined_metric_specification {
            predefined_metric_type = "ASGAverageCPUUtilization"
        }

        target_value = "${var.as_target_cpu_value}"
    }    
}

resource "aws_autoscaling_policy" "ondemand_instance_as_policy" {
    depends_on = [
        "aws_autoscaling_group.ondemand_instance_asg",
        "aws_launch_template.ondemand_instance_template"
    ]

    count                  = "${!var.use_spot_instance && var.use_auto_scaling_policy ? 1 : 0}"
    name                   = "${var.application}-${var.environment}-${var.instance_name}-as-policy"  
    policy_type            = "TargetTrackingScaling"    
    autoscaling_group_name = "${aws_autoscaling_group.ondemand_instance_asg.name}"

    target_tracking_configuration {
        predefined_metric_specification {
            predefined_metric_type = "ASGAverageCPUUtilization"
        }

        target_value = "${var.as_target_cpu_value}"
    }
}

resource "null_resource" "tags_as_list_of_maps" {
  count = "${length(keys(var.default_tags))}"

  triggers = {
    key                 = "${element(keys(var.default_tags), count.index)}"
    value               = "${element(values(var.default_tags), count.index)}"
    propagate_at_launch = "true"
  } 
}

Debug Output

panic: interface conversion: interface {} is nil, not map[string]interface {}
2019-03-11T10:28:25.739Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: 
2019-03-11T10:28:25.739Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: goroutine 83 [running]:
2019-03-11T10:28:25.739Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: github.com/terraform-providers/terraform-provider-aws/aws.readInstanceMarketOptionsFromConfig(0xc0008cf920, 0x3bd9681, 0x17, 0x2e8c200)
2019-03-11T10:28:25.739Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_launch_template.go:1360 +0x808
2019-03-11T10:28:25.739Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: github.com/terraform-providers/terraform-provider-aws/aws.buildLaunchTemplateData(0xc00092b0a0, 0x3c80d20, 0xc000af99c8, 0x0)
2019-03-11T10:28:25.739Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_launch_template.go:1071 +0x1624
2019-03-11T10:28:25.739Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsLaunchTemplateUpdate(0xc00092b0a0, 0x2e3bd20, 0xc000228c00, 0x24, 0x72db140)
2019-03-11T10:28:25.739Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_launch_template.go:677 +0x1ad
2019-03-11T10:28:25.740Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*Resource).Apply(0xc000592fc0, 0xc00066fe50, 0xc0005355e0, 0x2e3bd20, 0xc000228c00, 0x40b801, 0xc000af9b80, 0x4c1a0c)
2019-03-11T10:28:25.740Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/resource.go:231 +0x250
2019-03-11T10:28:25.740Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*Provider).Apply(0xc000247110, 0xc00066fe00, 0xc00066fe50, 0xc0005355e0, 0xc000056380, 0x18, 0x7fe923c836c0)
2019-03-11T10:28:25.740Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/provider.go:283 +0x9c
2019-03-11T10:28:25.740Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).Apply(0xc00031abc0, 0xc000535560, 0xc0009237d0, 0x0, 0x0)
2019-03-11T10:28:25.740Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/plugin/resource_provider.go:527 +0x57
2019-03-11T10:28:25.740Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: reflect.Value.call(0xc000075080, 0xc00000c030, 0x13, 0x3b9c1f7, 0x4, 0xc000af9f18, 0x3, 0x3, 0xc0005a8480, 0xc000136700, ...)
2019-03-11T10:28:25.740Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: 	/opt/goenv/versions/1.11.5/src/reflect/value.go:447 +0x454
2019-03-11T10:28:25.740Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: reflect.Value.Call(0xc000075080, 0xc00000c030, 0x13, 0xc00036b718, 0x3, 0x3, 0x5d798b, 0xc00036b728, 0x429852)
2019-03-11T10:28:25.740Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: 	/opt/goenv/versions/1.11.5/src/reflect/value.go:308 +0xa4
2019-03-11T10:28:25.740Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: net/rpc.(*service).call(0xc000070580, 0xc00008c5a0, 0xc0000365d8, 0xc0000368d0, 0xc0000f0000, 0xc0006abe40, 0x2e41060, 0xc000535560, 0x16, 0x2e410a0, ...)
2019-03-11T10:28:25.740Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: 	/opt/goenv/versions/1.11.5/src/net/rpc/server.go:384 +0x14e
2019-03-11T10:28:25.740Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: created by net/rpc.(*Server).ServeCodec
2019-03-11T10:28:25.740Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: 	/opt/goenv/versions/1.11.5/src/net/rpc/server.go:481 +0x47e
2019/03/11 10:28:25 [ERROR] root.umbraco.master_auto_scaler: eval: *terraform.EvalDiff, err: unexpected EOF
2019/03/11 10:28:25 [ERROR] root.umbraco.master_auto_scaler: eval: *terraform.EvalSequence, err: unexpected EOF
2019/03/11 10:28:25 [TRACE] [walkApply] Exiting eval tree: module.umbraco.module.master_auto_scaler.aws_launch_template.spot_instance_template
2019/03/11 10:28:25 [TRACE] root.umbraco.slave_auto_scaler: eval: *terraform.EvalWriteState
2019/03/11 10:28:25 [TRACE] root.umbraco.slave_auto_scaler: eval: *terraform.EvalApplyProvisioners
2019/03/11 10:28:25 [TRACE] root.umbraco.slave_auto_scaler: eval: *terraform.EvalIf
2019/03/11 10:28:25 [TRACE] root.umbraco.slave_auto_scaler: eval: *terraform.EvalWriteState
2019/03/11 10:28:25 [TRACE] root.umbraco.slave_auto_scaler: eval: *terraform.EvalWriteDiff
2019/03/11 10:28:25 [TRACE] root.umbraco.slave_auto_scaler: eval: *terraform.EvalApplyPost
2019/03/11 10:28:25 [ERROR] root.umbraco.slave_auto_scaler: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:

* aws_launch_template.spot_instance_template: unexpected EOF
2019/03/11 10:28:25 [ERROR] root.umbraco.slave_auto_scaler: eval: *terraform.EvalSequence, err: 1 error(s) occurred:

* aws_launch_template.spot_instance_template: unexpected EOF
2019/03/11 10:28:25 [TRACE] [walkApply] Exiting eval tree: module.umbraco.module.slave_auto_scaler.aws_launch_template.spot_instance_template
2019/03/11 10:28:25 [TRACE] dag/walk: upstream errored, not walking "module.umbraco.module.slave_auto_scaler.aws_autoscaling_group.spot_instance_asg"
2019/03/11 10:28:25 [TRACE] dag/walk: upstream errored, not walking "module.umbraco.module.slave_auto_scaler.aws_autoscaling_policy.spot_instance_as_policy"
2019/03/11 10:28:25 [TRACE] dag/walk: upstream errored, not walking "module.umbraco.module.slave_auto_scaler.aws_autoscaling_group.spot_instance_asg (destroy)"
2019/03/11 10:28:25 [TRACE] root.dynamic-xplan-proxy: eval: *terraform.EvalWriteState
2019/03/11 10:28:25 [TRACE] root.dynamic-xplan-proxy: eval: *terraform.EvalApplyProvisioners
2019/03/11 10:28:25 [TRACE] root.dynamic-xplan-proxy: eval: *terraform.EvalIf
2019/03/11 10:28:25 [TRACE] root.dynamic-xplan-proxy: eval: *terraform.EvalWriteState
2019/03/11 10:28:25 [TRACE] root.dynamic-xplan-proxy: eval: *terraform.EvalWriteDiff
2019/03/11 10:28:25 [TRACE] root.dynamic-xplan-proxy: eval: *terraform.EvalApplyPost
2019/03/11 10:28:25 [ERROR] root.dynamic-xplan-proxy: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:

* aws_security_group.ecs_service: unexpected EOF
2019/03/11 10:28:25 [ERROR] root.dynamic-xplan-proxy: eval: *terraform.EvalSequence, err: 1 error(s) occurred:

* aws_security_group.ecs_service: unexpected EOF
2019/03/11 10:28:25 [TRACE] [walkApply] Exiting eval tree: module.dynamic-xplan-proxy.aws_security_group.ecs_service
2019/03/11 10:28:25 [TRACE] dag/walk: upstream errored, not walking "module.dynamic-xplan-proxy.aws_ecs_service.service"
2019/03/11 10:28:25 [TRACE] dag/walk: upstream errored, not walking "provider.aws (close)"
2019/03/11 10:28:25 [TRACE] dag/walk: upstream errored, not walking "meta.count-boundary (count boundary fixup)"
2019/03/11 10:28:25 [TRACE] dag/walk: upstream errored, not walking "root"
2019-03-11T10:28:25.744Z [DEBUG] plugin: plugin process exited: path=/app/.terraform/plugins/linux_amd64/terraform-provider-aws_v2.1.0_x4
2019/03/11 10:28:25 [TRACE] Preserving existing state lineage "b7874485-b576-954e-d99f-ab791d3a9a0a"
2019/03/11 10:28:25 [TRACE] Preserving existing state lineage "b7874485-b576-954e-d99f-ab791d3a9a0a"
2019/03/11 10:28:25 [TRACE] Preserving existing state lineage "b7874485-b576-954e-d99f-ab791d3a9a0a"

Expected Behavior

The new auto scaling instances would successfully be created

Actual Behavior

panic

Steps to Reproduce

Issue is intermittent, 1 in 4 CI builds are successfully (with no changes made)

References

I couldn't find any.

@ghost ghost added bug Addresses a defect in current functionality. crash Results from or addresses a Terraform crash or kernel panic. labels Mar 11, 2019
@bflad bflad added the service/ec2 Issues and PRs that pertain to the ec2 service. label Mar 11, 2019
@JPScutt
Copy link
Author

JPScutt commented Mar 12, 2019

After looking through the panic error stack and the terraform provider code I removed the spot_options section from the terraform

instance_market_options {
market_type = "spot"

    **spot_options {
        max_price = "${var.spot_price}"
    }**        
}

var.spot_price was set to "". Reading through the aws provider code this should not be an issue

if v, ok := so["max_price"].(string); ok && v != "" {
spotOptions.MaxPrice = aws.String(v)
}

The error has not happened since removing that section

@nywilken
Copy link
Contributor

Relevant part of log

panic: interface conversion: interface {} is nil, not map[string]interface {}
2019-03-11T10:28:25.739Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: 
2019-03-11T10:28:25.739Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: goroutine 83 [running]:
2019-03-11T10:28:25.739Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: github.com/terraform-providers/terraform-provider-aws/aws.readInstanceMarketOptionsFromConfig(0xc0008cf920, 0x3bd9681, 0x17, 0x2e8c200)
2019-03-11T10:28:25.739Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_launch_template.go:1360 +0x808
2019-03-11T10:28:25.739Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: github.com/terraform-providers/terraform-provider-aws/aws.buildLaunchTemplateData(0xc00092b0a0, 0x3c80d20, 0xc000af99c8, 0x0)
2019-03-11T10:28:25.739Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_launch_template.go:1071 +0x1624
2019-03-11T10:28:25.739Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsLaunchTemplateUpdate(0xc00092b0a0, 0x2e3bd20, 0xc000228c00, 0x24, 0x72db140)
2019-03-11T10:28:25.739Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_launch_template.go:677 +0x1ad
2019-03-11T10:28:25.740Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*Resource).Apply(0xc000592fc0, 0xc00066fe50, 0xc0005355e0, 0x2e3bd20, 0xc000228c00, 0x40b801, 0xc000af9b80, 0x4c1a0c)
2019-03-11T10:28:25.740Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/resource.go:231 +0x250
2019-03-11T10:28:25.740Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*Provider).Apply(0xc000247110, 0xc00066fe00, 0xc00066fe50, 0xc0005355e0, 0xc000056380, 0x18, 0x7fe923c836c0)
2019-03-11T10:28:25.740Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/provider.go:283 +0x9c
2019-03-11T10:28:25.740Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).Apply(0xc00031abc0, 0xc000535560, 0xc0009237d0, 0x0, 0x0)
2019-03-11T10:28:25.740Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/plugin/resource_provider.go:527 +0x57
2019-03-11T10:28:25.740Z [DEBUG] plugin.terraform-provider-aws_v2.1.0_x4: reflect.Value.call(0xc000075080, 0xc00000c030, 0x13, 0x3b9c1f7, 0x4, 0xc000af9f18, 0x3, 0x3, 0xc0005a8480, 0xc000136700, ...)

@nywilken
Copy link
Contributor

Hi @JPScutt thanks for reporting this issue and for detailing your workaround. Looks like we might need a nil check or two in place.

@ryndaniels ryndaniels self-assigned this May 17, 2019
@ryndaniels
Copy link
Contributor

Hi @JPScutt - any chance you'd be able to test this using the most recent version of the AWS provider? I've so far been unable to reproduce this on 2.11.0 or 2.10.0, but if you're still seeing it on those versions that will be a good data point for us to start digging into more. Thanks!

@ryndaniels
Copy link
Contributor

Hey @JPScutt - I was just able to reproduce this crash on 2.11.0 so probably no need to test on your end if you haven't already - I'll continue looking into this now. Thanks!

@JPScutt
Copy link
Author

JPScutt commented May 20, 2019 via email

@bflad
Copy link
Contributor

bflad commented Jun 7, 2019

This has been released in version 2.14.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

@ghost
Copy link

ghost commented Nov 3, 2019

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators Nov 3, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Addresses a defect in current functionality. crash Results from or addresses a Terraform crash or kernel panic. service/ec2 Issues and PRs that pertain to the ec2 service.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants