user-data sometimes corrupted

[mcanevet]

Terraform Version

v0.9.4

Affected Resource(s)

• aws_instance

Terraform Configuration Files

data "template_cloudinit_config" "config" {
  gzip = true
  base64_encode = false
  ...
}

resource "aws_instance" "my_instance" {
  user_data = "${data.template_cloudinit_config.config.rendered}"
  ...
}

Expected Behavior

Cloud-init should always get the correct user_data

Actual Behavior

Somehow, sometimes cloud-init does not apply the configuration. When I log into an instance that failed and try to get the user_data, it looks like the metadata does not serves un gzipped content...

root@ip-10-0-0-43:~# curl -s http://169.254.169.254/2009-04-04/user-data > user_data.gz
root@ip-10-0-0-43:~# file user_data.gz 
user_data.gz: data
root@ip-10-0-0-43:~# gunzip user_data.gz 

gzip: user_data.gz: not in gzip format

If I taint the aws_instance resource and apply it again (without changing the template_cloudinit_config datasource it works...
I'm not sure if it's a bug in aws metadata or in terraform as I don't understand in details how user_data works.

Unfortunately I can't always reproduce the issue...

[practicalint]

I am having a similar problem. AWS User Data is just holding a simple text value of the hostname supplied from a variable. The variable is available and used for other purposes and is fine. Looks like it has been encrypted or something. When I look at the instance in AWS console it says this about the user data: An error occurred fetching user data (URIError) : URI malformed

resource "aws_instance" "TestNode" {
ami = "ami-xxxxxx"
instance_type = "${var.instance_type}"
user_data = "${var.instance_name}"
...
tags {
Name = "${var.instance_name}"
...
The tag comes out looking like expected.

An example of the user data as supplied to a script from the variable: CSP_HOSTNAME = q▒L▒▒޲▒hu▒5

From show command: user_data = 666579a558eaaf2ccc9100c6201d4713e3f6ad23

I changed backends and ran this fresh as I saw one issue that it could be a state issue, but that didn't help.

[majormoses]

This look like it has been outlined in the CHANGELOG under breaking changes: https://github.com/hashicorp/terraform/blob/master/CHANGELOG.md#094-26th-april-2017

[mcanevet]

@majormoses I don't think this is due to this change as the corruption does not always occurs and tainting and recreating the aws_instance resource fixes the issue when it occurs...

[majormoses]

hmm then this is a bug, I just upgraded to 0.9.5 and have not seen anything like this yet. Have you seen this happen with launch_config user_data as well? If I see anything that seems like this I will report back. We only use launch_config and auto_scaling groups so I will let you know if I see the same at some point with them.

[mcanevet]

@majormoses we are not using auto_scaling groups yet, so I can't tell if it works or not

[majormoses]

ok I will report back if I see it then, it might be limited to only aws_instance which we don't really use anymore.

[rajcheval]

I am seeing it in user_data for a launch configuration as well. We use data template_file to read and transform the user data. We are noticing that in 0.9.5 we see user data changes on every apply. I can understand why this happened once after upgrading from 0.9.4 to 0.9.5 but it is happening repeatedly on 0.9.5. I have worked around it by using using lifecycle ignore_changes = ["user_data"] for now.

[majormoses]

I have not seen this yet and the only difference that I have is that we are not using a template and passing in user data directly.

[tamsky]

FYI, I believe hashicorp/terraform-provider-aws#850 (merged) should fix the behavior you're seeing.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.