user-data sometimes corrupted

[hashibot]

This issue was originally opened by @mcanevet as hashicorp/terraform#14314. It was migrated here as part of the provider split. The original body of the issue is below.

---

Terraform Version

v0.9.4

Affected Resource(s)

• aws_instance

Terraform Configuration Files

data "template_cloudinit_config" "config" {
  gzip = true
  base64_encode = false
  ...
}

resource "aws_instance" "my_instance" {
  user_data = "${data.template_cloudinit_config.config.rendered}"
  ...
}

Expected Behavior

Cloud-init should always get the correct user_data

Actual Behavior

Somehow, sometimes cloud-init does not apply the configuration. When I log into an instance that failed and try to get the user_data, it looks like the metadata does not serves un gzipped content...

root@ip-10-0-0-43:~# curl -s http://169.254.169.254/2009-04-04/user-data > user_data.gz
root@ip-10-0-0-43:~# file user_data.gz 
user_data.gz: data
root@ip-10-0-0-43:~# gunzip user_data.gz 

gzip: user_data.gz: not in gzip format

If I taint the aws_instance resource and apply it again (without changing the template_cloudinit_config datasource it works...
I'm not sure if it's a bug in aws metadata or in terraform as I don't understand in details how user_data works.

Unfortunately I can't always reproduce the issue...

[apparentlymart]

Unfortunately it seems that this issue is caused by the fact that Terraform is not designed to preserve raw binary data stored in variables.

The reason it seems to work intermittently is that if the rendering of the cloudinit config and its use in the user_data occur in the same Terraform run, the data only ever exists in memory and thus although it is an invalid string Terraform just passes it on verbatim and it works. It fails any time Terraform does something that assumes it's a valid (UTF-8) string, which includes writing it to the state file and then reading it back again.

This difference can be seen in the plan output if I first allow it to deal with both resources at the same time:

+ local_file.baz
    content:  "\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xffd\xce\xc1J\x031\x10\xc6\xf1{ \xef\x10r\xcfVO\u0096\x1e\xd4\xf6\xe0a\x15\xa4\n\x1e'ͬ\x1dHfB2\xc5֧\x17\vb\xd1\xe3\xc0\u007f>~\xf7\u008a\xaca{\xaa8\xbar\xc8J\x15\x9a.\n\x1d1-]\x94\x03'h\xa7\x95\x9f\x1e\xa6\xcd\xdd\xd3\xcb\xe3\xfa\xf6\xf9͛\xef+\xbcb\xeb$<\xba\xeb\xe1\xca\x1akB\xb8\x8c\xac\xf9\xd9^S\xaf\xd2I\xcf-\xa8\xc2n_\x90u\xe9f\xca\xc8Pp\xe5g\x91!B\x1b\"|\xfa\xdf\xc7m\x03\xee3\xb6\xb0\xe1\x9d$\xe2\xf7\xd1\xddDҋ\xe0\xacV<\xea\xa2f \xb6f\xa2\x82\xff`{\xccY܇\xb4\x9c\xfe*C\xb0\xe6+\x00\x00\xff\xff\x1fy]\xc5\x04\x01\x00\x00"
    filename: "thing.gz"

...and then use -refresh=false to force Terraform to use the value stored in state:

-/+ local_file.baz (tainted)
    content:  "\x1f�\b\x00\x00\x00\x00\x00\x00�d�AK\x031\x10�������l�$l�Am\x0f\x1eVA��q��ځd&$S����\x16��e�\a�{|�\u008a�~;\x17\x1c\\>$�\x02U\x17��\x18�.ȁ#�yՍ\x0f�����q}��֙��_�6\x12\x1e�u\u007fe�5�_B��l��\x15i�'\x16Ta��Ⱥt\x13%dȸ�&�>@�\x03|v��m\x05n\x13V���D����\x04�\v�d�x�EI@l�H\x19���1%q\x1fRS���|�*{o�W\x00\x00\x00��\x1bh\x05M\x11\x01\x00\x00" => "\x1f�\b\x00\x00\x00\x00\x00\x00�d�AK\x031\x10�������l�$l�Am\x0f\x1eVA��q��ځd&$S����\x16��e�\a�{|�\u008a�~;\x17\x1c\\>$�\x02U\x17��\x18�.ȁ#�yՍ\x0f�����q}��֙��_�6\x12\x1e�u\u007fe�5�_B��l��\x15i�'\x16Ta��Ⱥt\x13%dȸ�&�>@�\x03|v��m\x05n\x13V���D����\x04�\v�d�x�EI@l�H\x19���1%q\x1fRS���|�*{o�W\x00\x00\x00��\x1bh\x05M\x11\x01\x00\x00"
    filename: "thing.gz" => "thing.gz"

Notice that in the second example there are lots of Unicode Replacement Character (�) which results when attempting to interpret non-UTF8 data as UTF8.

Eventually we will hopefully have a way to use raw binary data in Terraform, but in the short term I'm planning to deal with this in the following way:

• Add a new user_data_base64 argument to both aws_instance and aws_launch_configuration that accepts input that is base64 encoded and decodes it before writing it to the EC2 API. This new argument will be an alternative to (and in conflict with) user_data.
• Make template_cloudinit_config disallow gzip without base64_encode, since that combination results in this corruption.

The example in the original comment would then become:

data "template_cloudinit_config" "config" {
  gzip = true
  base64_encode = true
  ...
}

resource "aws_instance" "my_instance" {
  user_data_base64 = "${data.template_cloudinit_config.config.rendered}"
  ...
}

Since the EC2 resource implementations will decode the base64 encoding client-side, this will achieve an equivalent result but will avoid the risk of data corruption caused by trying to pass the binary data between resources directly.

This is consistent with (and complementary to) where we landed on hashicorp/terraform#3858, which would've suffered the same problem.

[mcanevet]

@apparentlymart thanks for your comprehensive explanation (as usual).

[tamsky]

I think this bug should be almost fixed and can be closed upon the next release of the AWS provider?

#497 (comment)

The new attribute [user_data_base64] is not yet included in a stable release, but it should be included in the next one.

[Ninir]

Closing as per #497 (comment)

[Ninir]

Thanks for pointing this out @tamsky !

[jreyeshdez]

I am getting exact same issue if I use aws_launch_template

data "template_cloudinit_config" "config" {
  gzip = true
  base64_encode = true

  part {
    filename     = "test"
    content_type = "text/cloud-config"
    content      = "${file("${path.module}/../../resources/test")}"
  }
}

resource "aws_launch_template" "template" {
  ...
  user_data = "${data.template_cloudinit_config.config.rendered}"
  ...
}

Worth mentioning my cloud-init file has #cloud-config at the top of my resources/test.

Unfortunately, user_data_base64 is not available in aws_launch_template.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!