[Bug]: Existing tags being reapplied to resources when there are no changes.

[iodeslykos]

Terraform Core Version

1.5.7

AWS Provider Version

5.41.0

Affected Resource(s)

aws_ec2_instance and its sub-resource root_block_device.

Expected Behavior

No plan should be generated, since there are no changes to apply.

Actual Behavior

Terraform generates a plan that wants to add tags that already exist to resources.

Relevant Error/Panic Output Snippet

aws_instance.ec2 will be updated in-place
  ~ resource "aws_instance" "ec2" {
        id                                   = "<redacted>"
        tags                                 = {
            "Tag1"      = "<redacted>"
            "Tag2"     = "<redacted>"
            "Tag3"    = "<redacted>"
            "Tag4"      = "<redacted>"
            "Tag5" = "<redacted>"
            "Tag6"    = "<redacted>"
            "Tag7"       = "<redacted>"
            "Tag8"         = "<redacted>"
            "Tag9"   = "<redacted>"
            "Tag10"         = "<redacted>"
            "Tag11"              = "<redacted>"
            "Tag12"        = "<redacted>"
            "Tag13"          = "<redacted>"
            "Tag14"           = "<redacted>"
            "Tag15"   = "<redacted>"
            "Tag16"     = "<redacted>"
            "Tag17"              = "<redacted>"
            "Tag18"       = "<redacted>"
        }
        # (30 unchanged attributes hidden)

      ~ root_block_device {
          ~ tags                  = {
              + "Tag1"      = "<redacted>"
                "Tag2"      = "<redacted>"
              + "Tag3" = "<redacted>"
              + "Tag4"    = "<redacted>"
                "Tag5"       = "<redacted>"
                "Tag6"      = "<redacted>"
              + "Tag7"         = "<redacted>"
                "Tag8"              = "<redacted>"
              + "Tag9"           = "<redacted>"
              + "Tag10"   = "<redacted>"
              + "Tag11"     = "<redacted>"
              + "Tag12"              = "<redacted>"
              + "Tag13"       = "<redacted>"
                "Tag14"        = "<redacted>"
            }
            # (10 unchanged attributes hidden)
        }

Terraform Configuration Files

# ec2.tf

locals {
  instance_locals = {
    Name            = var.hostname
    AnsibleManaged  = "true"
    AnsibleGroups   = var.ansible_groups
    BackupPolicy    = var.backup_policy
    Platform        = "<redacted>"
    PatchGroup      = var.patch_group
    ImageName       = data.aws_ami.ami.name
    ImageNamePrefix = length(regexall("^(.*)[-\\d]{16}$", data.aws_ami.ami.name)) > 0 ? regex("^(.*)[-\\d]{16}$", data.aws_ami.ami.name)[0] : data.aws_ami.ami.name
  }

  volume_locals = {
    Name         = var.hostname
    VolumeName   = var.hostname
    InstanceName = var.hostname
    BackupPolicy = var.backup_policy
  }

  tags        = merge(var.tags, local.instance_locals, var.extra_tags)
  volume_tags = merge(var.tags, var.volume_tags, local.volume_locals, var.extra_tags)
}

resource "aws_instance" "ec2" {
  ami                     = data.aws_ami.ami.id
  instance_type           = var.instance_type
  subnet_id               = data.aws_subnet.selected.id
  disable_api_termination = var.disable_api_termination
  iam_instance_profile    = var.instance_profile
  vpc_security_group_ids  = var.security_group_ids
  tags                    = var.tags
  ebs_optimized           = var.ebs_optimized
  ipv6_address_count      = var.ipv6_address_count
  key_name                = var.key_name

  root_block_device {
    volume_size           = var.root_size
    volume_type           = var.root_type
    delete_on_termination = var.root_termination
    encrypted             = var.encrypted
    tags                  = var.volume_tags
  }

# provider.tf

provider "aws" {
  region = var.region

  assume_role {
    role_arn = "arn:aws:iam::111111111111:role/admin_role"
  }
}

# terraform.tf

terraform {
  required_version = ">= 1.5"

  backend "s3" {}

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.40"
    }

# terraform.tfvars

tags = {
  Tag1  = "<redacted>"
  Tag2  = "<redacted>"
  Tag3  = "<redacted>"
  Tag4  = "<redacted>"
  Tag5  = "<redacted>"
  Tag6  = "<redacted>"
  Tag7  = "<redacted>"
  Tag8  = "<redacted>"
  Tag9  = "<redacted>"
  Tag10 = "<redacted>"
  Tag11 = "<redacted>"
  Tag12 = "<redacted>"
  Tag13 = "<redacted>"
  Tag14 = "<redacted>"
  Tag15 = "<redacted>"
  Tag16 = "<redacted>"
  Tag17 = "<redacted>"
  Tag18 = "<redacted>"
}

Steps to Reproduce

Create an aws_instance resource with tags applied to the root_block_device. Run multiple times.

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[jpriebe]

I believe this is specific to aws_instance.root_block_device. We recently started seeing this issue (maybe 4 weeks ago?) on all our root block devices. I suspect that a new AWS provider release happened that introduced the bug (maybe https://github.com/hashicorp/terraform-provider-aws/releases/tag/v5.39.1, which specifically mentions root_block_device and tags_all)

We are heavy users of default_tags, and we have to jump through hoops to get the default tags onto EC2 instances' root block devices, so I'm particularly sensitive to anything related to tags on root block devices.

Terraform version 1.2.9
AWS provider version 5.41.0

[jpriebe]

I wonder if this behavior stems from #33769, which finally fixes the issue of default_tags not being applied to root_block_device.

Just for background, we set a number of default tags in our provider. In order to get them onto the root block device, we have to do something like this:

data "aws_default_tags" "current" {}

locals {
  all_tags = data.aws_default_tags.current.tags
}

resource "aws_instance" "foo" {
   ...

  root_block_device {
    volume_size = 30
   ...
    tags = merge({
      Name = "foo"
    }, local.all_tags)
  }

I wonder if the fact that the tags are present both in the default_tags and the explicit tags that we are adding to root_block_device is causing the problem?

[sahaqaa]

Hello, we have the same issue - every time we run "terraform plan" or "terraform apply" Terraform wants to change tags:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.oleksandr-lytvyn-test.aws_instance.ec2_instance[0] will be updated in-place
  ~ resource "aws_instance" "ec2_instance" {
        id                                   = "i-XXXXXXXXXX"
        tags                                 = {
            "Name"              = "oleksandr-lytvyn-test
            "config-management" = "XXX"
            "backup"        = "no-backup"
        }
      ~ volume_tags                          = {
          + "created-by"  = "XXX"
          + "environment" = "staging"
          + "maintainer"  = "XXX"
          + "owner"       = "XXX"
          + "task-group"  = "XXX"
            # (1 unchanged element hidden)
        }
        # (33 unchanged attributes hidden)

        # (8 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

To be more specific about our use case:
1/ We have our custom module that creates EC2 instance and adds our custom logic
2/ We have AWS Organizations SCP applied that prevents from creation of EC2 and EBS Volumes without specific required tags.

Prior to AWS Terraform provider version 5.39.0 everything was OK. For example with 5.38.0, 5.37.0, 5.36.0 etc -> when we created EC2 via our custom module -> EC2 was created with "default_tags" and EBS Volume was created with all required tags. Logic to assign "volume_tags" that we use is next:

data "aws_default_tags" "current" {}

resource "aws_instance" "ec2_instance" {
....
volume_tags = merge(data.aws_default_tags.current.tags, var.tags, )
....
}

Important note: we use "volume_tags" and we do NOT touch "root_block_device" section in any way (regarding tags):

  root_block_device {
    volume_type = var.volume_type
    volume_size = var.volume_size
    encrypted   = var.is_ebs_root_volume_encrypted
    kms_key_id  = var.is_ebs_root_volume_encrypted == true ? local.kms_key_arn : null
  }

---

We assign "default_tags" like this:

provider "aws" {
  region = "us-west-1"
  default_tags {
    tags = {
      maintainer = "XXX"
      owner       = "XXX"
      task-group  = "XXX"
      environment = "XXX"
      created-by  = "XXX"
    }
  }
}

Terraform version:

Terraform v1.6.6
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v5.42.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.3.3

After release of AWS Terraform provider version 5.39.0 (and up to latest one, which is 5.42.0 at this moment) - every time we run "terraform plan" and "terraform apply" Terraform wants to update in-place "volume_tags" (despite the fact that "volume_tags" already in good state.

Please advice - is there any way to get this fixed?

[oben59]

I confirm the behavior with default_tags and the root_block_device, I have it on my end as well: perpetual change. This only seem to happen for tags that are present in the default and the root_block_device tags. This is done dynamically for any evaluation of the root_block_device tags, even if you use volume_tags

I searched in the code and found these lines:

terraform-provider-aws/internal/service/ec2/ec2_instance.go

Lines 1299 to 1301 in 7e31a73

	if err := d.Set("volume_tags", tags.RemoveDefaultConfig(defaultTagsConfig).Map()); err != nil {
	return sdkdiag.AppendErrorf(diags, "setting volume_tags: %s", err)
	}

I'm unsure about the logic expressed here so I can't produce a fix for now, this seems to force the default_tags to always be missing from the in memory representation of the instance before evaluation of the changes, hence the perpetual change when compared to the actual desired tags for the block.

[n2N8Z]

We have perpetual change on volume_tags too. We also use default_tags. This only started happening recently, maybe in the last month or two.

[erlarson85]

This is really really painful when you implemented default tags for all of your infrastructure. Every pipeline run requires review and approval.

[0xkag]

I opened PR #37441 for this. We encountered this problem and this patch fixes it for us. We've been running a local build of the provider because the false drift was unbearable. It's a pretty simple patch, but it needs a maintainer to look at it of course.

[iodeslykos]

@0xkag Our savior. Thank you.

[github-actions]

Warning

This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[github-actions]

Warning

This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[github-actions]

This functionality has been released in v5.72.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.