Attached EBS Volume tags flip flop between volume_tags for instance on each apply

[krystan]

I came across this today,

It looks like this is a well-known issue, it is detailed here:

hashicorp/terraform-provider-aws#770

If I have EBS volumes with tags which are specified as attached to the instances then the tags get clobbered in a circular fashion.

There is a current workaround mentioned in the above thread but this for the moment means the correct tagging of EBS volumes attached to instances generated by this module is not possible as it will change alternately on each Apply.

[rba1-source]

To get them to flip-flop, run the terraform apply, then run a plan and you'll see it's trying to change the instance volume_tags and the tags on the volumes. Then you can run a second apply, followed by a plan, and you'll see it now wants to do the reverse.

Output for flop 1 is:

  # module.storage.module.primary.aws_ebs_volume.volume_output__backup will be updated in-place
  ~ resource "aws_ebs_volume" "volume_output__backup" {
      ~ tags              = {
          + "Backup"                 = "true"
          + "Environment"            = "dev"
            "Name"                   = "storage-primary-dev-volume_output__backup"
          + "Team"                   = "myteam"
        }
    }

  # module.storage.module.primary.module.storage.aws_instance.this[0] will be updated in-place
  ~ resource "aws_instance" "this" {
        tags                         = {
            "Backup"                 = "true"
            "Environment"            = "dev"
            "Name"                   = "storage-primary-dev"
            "Role"                   = "storage_server"
            "Team"                   = "myteam"
        }
      ~ volume_tags                  = {
          ~ "Name" = "storage-primary-dev-volume_output" -> "storage-primary-dev"
        }

Output for flop 2 is:

  # module.storage.module.primary.aws_ebs_volume.volume_output__backup will be updated in-place
  ~ resource "aws_ebs_volume" "volume_output__backup" {
      ~ tags              = {
            "Backup"                 = "true"
            "Environment"            = "dev"
          ~ "Name"                   = "storage-primary-dev" -> "storage-primary-dev-volume_output__backup"
            "Team"                   = "myteam"
        }
    }

  # module.storage.module.primary.module.storage.aws_instance.this[0] will be updated in-place
  ~ resource "aws_instance" "this" {
        tags                         = {
            "Backup"                 = "true"
            "Environment"            = "dev"
            "Name"                   = "storage-primary-dev"
            "Role"                   = "storage_server"
            "Team"                   = "myteam"
        }
      ~ volume_tags                  = {
          - "Backup"                 = "true" -> null
          - "Environment"            = "dev" -> null
            "Name"                   = "storage-primary-dev"
          - "Team"                   = "myteam" -> null
        }
    }

I don't specify my ebs volumes in with the instance resource, I have separate aws_ebs_volume resources and I attach them using aws_volume_attachment resources.

I'm not specifying any value for volume_tags when calling the terraform-aws-ec2-instance module, because I want each attached volume to have its own set of tags (because the Name tag will be different for each one). The fields I use are:

1. "tags" when calling the terraform-aws-ec2-instance module
2. "tags" on the "aws_ebs_volume" resources

You can see a few odd things:

1. the flip-flopping, obviously
2. the volume_tags "Name" on the instance output; I am attaching multiple volumes and it looks like the volume_tag "Name" field has one of them (which, in this case, matches one of the other volumes, not the one I've displayed in the output). Why does the plan say that the Name tag in volume_tags has a value which matches one of my attached volumes? If I check in my statefile I do indeed just have 1 tag in volume_tags, not all of them (not that I want any in here):

    {
      "module": "module.storage.module.primary.module.storage",
      "mode": "managed",
      "type": "aws_instance",
      "name": "this",
      "each": "list",
      "provider": "provider.aws",
      "instances": [
        {
            "tags": {
              "Backup": "true",
              "Environment": "dev",
              "Name": "storage-primary-dev",
              "Role": "storage_server",
              "Team": "myteam"
            },
            "volume_tags": {
              "Name": "storage-primary-dev-volume_output__backup"
            },

I don't want to do a workaround of ignoring changes in the tags, because they may change in future.

Ideally, if I don't specify any volume_tags, then I don't want it to add any, because I'm handling them separately.

[strowi]

Hi,

running into the same issue, even when no volume_tags are defined for the ec2 instance..
This is especially annoying e.g. if we attach multiple volumes, and want to backup only specific ones with a tag "backup".
Any solution to this?

[oivwneoisjgsdf]

Same issue here

[JamesTimms]

Hi I ran into this today too. The below should fix this issue but I don't know if it would introduce any unwanted side-effects.

  lifecycle {
    ignore_changes = [
      # Prevents clobbering the tags of attached EBS volumes
      volume_tags,
    ]
  }

[JamesTimms]

It seems the cleanest workaround is to define tags outside of the module for now.
hashicorp/terraform-provider-aws#12226 (comment)

resource "aws_ec2_tag" "root_volume_tags" {
  # Workaround for known bug, https://github.com/terraform-aws-modules/terraform-aws-ec2-instance/issues/164
  for_each = {
    Name   = "VALUE"
    ANOTHER_TAG = "ANOTHER_VALUE"
  })

  resource_id = module.ec2_instance.root_block_device_volume_ids[0][0]
  key         = each.key
  value       = each.value
}

[nigelellis]

This appears fixed in AWS 3.24.0+ with the addition of tags as a param on the root_block_device. See hashicorp/terraform-provider-aws#15474.

[nikowatari]

I don't use volume_tags, but after updating to 3.33.0 from 3.18.0 it showed in plan that volume_tags were nulled and partially deleted tags on my volumes as a result. it completely ignores tags on attached ebs_volumes (which wasn't the case before updating)

[PePoDev]

Still issue, Any idea to solve this ?

resource aws_ec2_tag did't solve this issue now

IMO, This module should remove volume_tags and manage on root_block_device or create option to toggle this.

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/instance#volume_tags

[syed-awais-ali]

I opened an issue since this problem has been bugging us as well. #216

[github-actions]

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

[github-actions]

This issue was automatically closed because of stale in 10 days

[richard-browne]

I guess it's not worth taking the time to report terraform bugs. They are ignored then closed because they are "stale".

[gmaz42]

@richard-browne perhaps we need to have regular weekly chats on the bugs we follow so that they do not get automatically closed, but that would also dilute the actual useful content in them 🤔 so there is an obvious bias for inaction here.

[a-schaefers]

Terraform shouldn't allow this to happen, it's more of a Terraform bug likely than a bug of this module.

When you put tags on an external EBS volume, you need to also have the same tags and values in the volume_tags attribute on the parent ec2 resource (whether you're using this module, or not.) See also https://stackoverflow.com/questions/62470272/how-can-i-add-a-tag-to-aws-ebs-when-creating-through-ec2-with-terraform

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.