Leftover module module.iam in state that should have been removed; symlink related?

[AaronSofaer-PGE]

Terraform Version

[Container] 2019/05/30 21:14:39 Running command terraform version 
Terraform v0.12.0 

Additional Context

Unfortunately, I am very limited in the amount of code, configuration, or environment data I can share without getting permission, which I can do if I can assert it's definitely necessary. (If it's not, it's not, hooray.)

This is a multi-stage Amazon Code Pipeline with multiple Build stages. Each one at the moment passes **/* as artifacts to the next, which is to say, everything. Within the repository's terraform directory there are code and pipeline subdirectories. Within the code subdirectory there are the usual .tf files and modules directory; within the pipeline subdirectory there are the usual .tf files and a symlink to the modules directory in the code subdirectory.

The purpose of this was to reduce the amount of repeated code, since some of the IAM role definitions were being used in both places, and more are going to be added that are identical between the two (Lambdas-related, mostly). I expect that having all of the modules in one place will improve maintainability of our configuration.

It's possible this is just straight-up unsupported, in which case I'll move on with my life with a more typical configuration, but I didn't see anything about it in the docs. (Also I hope that the bug report will be useful regardless.)

Expected Behavior

terraform apply functions cleanly and creates the desired resource.

Actual Behavior

·[1m·[31mError: ·[0m·[0m·[1mleftover module module.iam in state that should have been removed; this is a bug in Terraform and should be reported·[0m

Steps to Reproduce

(build is kicked off by pushing code or doing terraform apply from the pipeline folder, so this is running on the hashicorp light docker container)
cd terraform/code
terraform init --input=false -backend-config=env/backendConfig.tfvars
terraform apply -target=module.[redacted].[redacted] --var-file=env/dev.tfvars --input=false --auto-approve

[apparentlymart]

Hi @AaronSofaer-PGE! Sorry for this odd behavior.

Do you get the same error if you run terraform apply without that -target argument? I wonder if the target is inadvertently removing from the graph a node that would normally be responsible for cleaning up that module. If that is true, we may be able to fix it by adjusting the -target behavior to include whatever node that is.

Usually we need either a full trace log or a minimal reproduction case in order to be able to debug a problem like this, because otherwise the space of possible causes is too large to search. I understand that your full configuration is sensitive, and indeed it sounds like it's a pretty large configuration that would be hard for us to use for reproduction outside of your environment anyway.

If you're able to run Terraform with the TF_LOG=trace environment variable set and share the result of that (e.g. via gist) then that's probably the best way to get some insight into the problem here. If the log itself includes sensitive information, feel free to encrypt it using the HashiCorp security public key

[AaronSofaer-PGE]

@apparentlymart It did happen if I ran terraform apply without the -target argument. I wound up splitting out the IAM stuff and now that I've gotten that into a working state I can't reproduce the problem anymore, so my suspicion is that it had something to do with the two flows (code/ and pipeline/) both trying to create the same resource on the same terraform ... bucket? lock table in dynamoDB, whatever the correct term is for that.

I'll keep an eye out for it happening again and pass on the request to the team to keep an eye open for a recurrence, and we'll grab more context if it happens again, but for now if you want to close this I wouldn't object. ("I reported the bug and now can't reproduce it", oh god, I'm that guy.)

[apparentlymart]

Thanks for the extra context, @AaronSofaer-PGE!

No worries on not being able to reproduce it anymore. That happens! Indeed, I think as you said I'm going to close this out for now (since it's unlikely we'll be able to reproduce it without some additional information) and we can always start a new issue if you or someone else hits the same problem and has more information to share.

Thanks again for reporting this, and sorry we weren't able to get to the bottom of it this time.

[alext]

@apparentlymart I've come up with a minimal reproduction case for this, which I've posted on the related issue here: #21313 (comment)

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.