aws_volume_attachment shouldnt have to wait for instance to be running to attach

[gtmtech]

Terraform 0.9.8

I have an aws_instance terraformed, and in the 'stopped' state (by design). I should easily be able to attach an ebs volume to it in this state with an aws_volume_attachment. However terraforming an aws_volume_attachment to the instance leads to this error:

1 error(s) occurred:

* aws_volume_attachment.foo: 1 error(s) occurred:

* aws_volume_attachment.foo: Error waiting for instance (i-11112222333344445) to become ready: unexpected state 'stopped', wanted target 'running'. last error: %!s(<nil>)

This seems to be bad terraform design behaviour - the volume attachment should work just fine - indeed it is often safer to attach volumes to a stopped instance than to a started one.

[CpuID]

+1, this also causes race conditions when you expect a volume to be attached on a new aws_instance boot but it isn't there for a good 60~ seconds or so initially. The aws_instance creates first, then the aws_ebs_volume then finally the aws_volume_attachment.

The only workaround I have come up with so far is to use ebs_block_device within the aws_instance instead of this resource type.

[radeksimko]

cc @bodgit as author of hashicorp/terraform#12459

Instead of waiting for any particular state we could just wait until instance exists and isn't in terminated state - I assume that's the only state which doesn't allow volumes to be attached based on the above conversation.

[bodgit]

Instead of waiting for any particular state we could just wait until instance exists and isn't in terminated state - I assume that's the only state which doesn't allow volumes to be attached based on the above conversation.

Looking at my original code change, I would say the state should be something that is neither terminated nor pending, that would probably be sufficient to fix this.

[radeksimko]

@bodgit as far as I understand @CpuID 's reaction he says it's possible to attach a volume while the instance is in pending state - correct me if I'm wrong.

[bodgit]

That wasn't my experience. According to my remarks in the referenced PR, waiting for the spot instance request to be fulfilled meant the spot instance is then created in state pending but the volume attachment fails at least until it enters state running, (or for the sake of argument, leaves state pending), hence that PR. Maybe it's specific to spot instances?

[CpuID]

@bodgit as far as I understand @CpuID 's reaction he says it's possible to attach a volume while the instance is in pending state - correct me if I'm wrong.

I could be wrong there, I know ebs_block_device on the aws_instance allows you to have the EBS attached on first boot, whereas I believe aws_volume_attachment uses a separate non-atomic API call to attach after RunInstances, correct? I can't verify if you can attach during a pending from my testing so far.

[gtmtech]

@CpuID @radeksimko It is not possible (at least not always possible) to attach a volume to an instance in pending state. See for example this pastebin https://pastebin.com/LAc6fyur which I created which will reproduce the problem. The error it gives is:

An error occurred (IncorrectState) when calling the AttachVolume operation: Instance 'i-xxxxxxxxxxxxxxxxx' is not 'running'.

So the API clearly does not allow it - but the error message is also misleading, as the instance does not have to be running state, it just has to not be in pending, shutting down, terminating, etc. states, both stopped and running is fine

[bodgit]

In which case it should be trivial to just add stopped to the list of targets in the attachment code. Spot instances can't be stopped without terminating them so I never really tested that particular path.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!