aws_spot_instance_request failure causes dependent resources to block further use of terraform

[matthewfranglen]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.13.4

• provider registry.terraform.io/-/aws v3.11.0
• provider registry.terraform.io/hashicorp/aws v2.70.0

Affected Resource(s)

• aws_spot_instance_request
• aws_ec2_tag
• aws_volume_attachment

Terraform Configuration Files

resource "aws_spot_instance_request" "my_request" {
  wait_for_fulfillment = true
  spot_type = "one-time"
  key_name = var.ssh_key_name

  ami = var.elasticsearch_ami_id
  availability_zone = var.elasticsearch_availability_zone
  subnet_id = var.subnet_ids[var.elasticsearch_availability_zone]
  spot_price = "0.000001"
  instance_type = "t2.xlarge"
}

resource "aws_ec2_tag" "my_tag" {
  resource_id = aws_spot_instance_request.my_request.spot_instance_id
  key = "Name"
  value = "my-instance"
}

Debug Output

Panic Output

Expected Behavior

terraform state list should not contain aws_spot_instance_request.my_request
terraform destroy should not fail with an error.
terraform apply should not fail with an error.

Actual Behavior

When creating the spot instance the apply fails with the error:

}) to resolve: unexpected state 'price-too-low', wanted target 'fulfilled'. last error: %!s(<nil>)

  on instance.tf line N, in resource "aws_spot_instance_request" "my_request":
 N: resource "aws_spot_instance_request" "twitter_lists_elasticsearch" {

Once this error has occurred subsequent use of the terraform apply and terraform destroy commands fail with the error:

Error: "resource_id": required field is not set

This error is produced for the resource "aws_ec2_tag" "my_tag".
The failure means that destroy and apply cannot complete.

The resource for the spot instance has been created and can be viewed with terraform state list.
Running terraform state rm aws_spot_instance_request.my_request will fix the state and allow apply and destroy to work.

Steps to Reproduce

1. Make a spot instance request with a price that is too low to be fulfilled.
   2a. Perform terraform state list, see resource for spot instance request.
   2b. Perform terraform destroy, see error.
   2c. Perform terraform apply, see error.

Important Factoids

References

[gdavison]

Hi @matthewfranglen, thanks for raising this issue. It looks like there are several issues coming together here.

Firstly, there's an open issue on the Terraform Plugin SDK which is causing state to be written even when there's an error.

Secondly, when wait_for_fulfillment is true, we currently only support a limited set of the possible states of a Spot request. Specifically, we don't support the "Holding" states. These should probably be added.

Lastly, if the Spot request doesn't launch an instance, there will be no value for aws_spot_instance_request.my_request.spot_instance_id. However, resource_id is required for aws_ec2_tag. The best way around this is to only create the tag resource if there is an instance. Something like the following could work:

resource "aws_ec2_tag" "my_tag" {
  count       = aws_spot_instance_request.my_request.spot_instance_id == "" ? 0 : 1
  resource_id = aws_spot_instance_request.my_request.spot_instance_id
  key         = "Name"
  value       = "my-instance"
}

[gdavison]

I've created #15674 to address the states

[matthewfranglen]

Thanks for the tip about using count. I'll give that a go. Please let me know if I can be of any more help.

[matthewfranglen]

The count approach doesn't really work out unfortunately. This error occurs for both apply and delete:

The "count" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the count depends on.

As a workaround it's easier to just fix the state with terraform state rm aws_spot_instance_request.my_request when the error happens than run multiple apply commands. It was still a good suggestion.

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.