Hang during terraform plan in 0.6.7 #4043
Comments
|
Hi @landro - this is the same crash reported in #4036 and elsewhere that we believe is fixed in #4024. We're drawing a circle around a few important bugfixes and we'll be looking at a bugfix release soon. In the meantime, if you're able to check latest master to see if it solves your issue, that'd be helpful! |
|
I built fra master, and gave it a go. Now it hangs again (instead of crashing). I've attached debug logs |
|
Okay, so the key portion of the hanging debug log is here: Those three lines will likely continue to repeat every 5s while it hangs. Will dig in some more and follow up. |
|
Can you share the config for your |
I'm setting TF_VAR_access_key and TF_VAR_secret_key, reading into variable, and passing them to the aws provider like so |
|
Great thank you. Going to see if I can cook up a reproduction with this. |
phinze
changed the title
|
Our initial attempts to reproduce have not been successful. Does Terraform v0.6.6 still complete a plan successfully in your environment? |
|
Another shot in the dark to try out would be to switch from using Sorry for the trouble! We'll get to the bottom of this. |
|
So I tried running a plan using 0.6.6 (and the original provider config), and that also hangs for a long time, repetedly showing 2015/11/25 08:27:32 [DEBUG] vertex aws_route53_record.api, waiting for: provider.aws for like 5 mins and the it moves on, before it repeatedly shows the same thing for another 5 mins a few more times. In totalt, 20 minutes pass before plan is completed. Debug log attached. Grep-ing for "waiting for: aws_route53_record.api" in the log, shows it appearing every 5 sec during the entire run. I executed terraform plan one more time - again it took 20 minutes. |
|
Anywhere I could add debugging code in order to shed light on what's going on? |
|
I tried configuring the aws provider like so: That also triggers waiting for: provider.aws However: immediately fails with: Not a valid region: eu-central-9 |
|
I have the same problem, only i am not using route53. |
|
@dverbeek84 - have you tried building from master? |
|
@landro yes, i just did. It was not working. It also hangs. |
|
Switching to AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_DEFAULT_REGION resolved the issue for me. Originally my access credentials using variables and TF_VAR_ environment variables similar to @landro. I was experiencing this issue while running 'terraform plan' or 'terraform refresh'. All the aws resources would just repeatedly say they were waiting on another resource every 5 seconds. I tried three different versions, but none of them worked. v0.6.6 (hang), v0.6.7 (crash) and master (hang) |
|
Hey folks, some of the occurrences here might be explained by the HCL issue we found in #4082, which caused config parsing to hang on any file that ended in a comment line. That should be fixed on master with a |
|
I think such bug is rather critical and would be great if new version of Terraform is released (at least it should contain just this fix, if possible). Kind of, version 0.6.7.1 :) |
|
so removed trailing # and set credentials using AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_DEFAULT_REGION - and everything started working both in master and 0.6.6 |
|
@antonbabenko agreed! We're cutting a release today. :) |
|
I'll try to reproduce in a simple repo later today, @phinze |
|
Thanks @landro! |
|
Managed to reproduce issue with a dead simple tf-file HTH in tracking down the cause of this issue, @phinze |
|
This is super helpful, @landro. Will dig into this tomorrow. |
|
Hello again, unfortunately I'm not having any luck reproducing this using your example quite yet. I can plan, apply, refresh, and destroy using v0.6.8. I also tried v0.6.6 and v0.6.7 and was unable to reproduce any hang. The r53 APIs are quite slow - often taking minutes to complete creation, but I've gotten no hangs. My prereqs config executed separately to get a valid zone ID: variable "my_access_key" {}
variable "my_secret_key" {}
provider "aws" {
access_key = "${var.my_access_key}"
secret_key = "${var.my_secret_key}"
region = "eu-central-1"
}
resource "aws_vpc" "primary" {
cidr_block = "10.1.2.0/24"
}
resource "aws_route53_zone" "primary" {
vpc_id = "${aws_vpc.primary.id}"
name = "example.com"
}
output "zone_id" {
value = "${aws_route53_zone.primary.id}"
}The repro config: variable "my_access_key" {}
variable "my_secret_key" {}
provider "aws" {
access_key = "${var.my_access_key}"
secret_key = "${var.my_secret_key}"
region = "eu-central-1"
}
resource "aws_route53_record" "default" {
zone_id = "Z3TP2LAVI4BAXL" # Output from prereqs config
name = "example.com"
type = "A"
ttl = "300"
records = [
"10.0.0.1"
]
}Other details:
Questions for @landro and anybody else seeing hangs:
|
|
Hi guys, I experienced the same for both v0.6.7 and v0.6.8. After a little bit of debugging and network tracing it turned out that the reason it hangs and loops forever is the fact that AWS provider does some background checks and tries to connect On AWS machines response to As soon as I added a rule that drops all connections to |
|
After upgrading to 0.6.7 it works fine on OS X for me. |
|
Hey @landro – are you running this on your local machine, or from an EC2 instance? |
|
Sorry for the long wait. Just to clarify @phinze - I manage to reproduce the issue using without any prereqs. I run the above command as is - literally using acc as my access key and sec as my secret key, and without modifying the bug.tf file in any way. I have run this from my laptop @catsby, on different networks, home, two different company networks and mobile 4g network. Same hangs all over. I tried looking into what's going on with the 169.254.169.254:80 port that @jwadolowski mentioned. It turns out, it accepts connections even when I'm disconnected from the internet (I take down my wifi connection). nmap -sT -p80 169.254.169.254 Nmap done: 1 IP address (1 host up) scanned in 0.03 seconds I believe, it might be a good idea to find a better way to sort out if we're running terraform from amazon or not. BTW, in my current project, we're using a private cloud running cloudstack, and only using aws for dns. |
|
@landro this is very good information. We had an internal chat about this issue and #3243 + #2693 and decided to revamp AWS creds handling in the provider. @catsby explains more in his comment over here: #2693 (comment) Stay tuned for a new PR that hopefully addresses all these issues! 👍 |
|
Hello friends! If any of you could be so kind as to build from |
|
Hey all – I'm going to go ahead and close this, please let us know if you're still hitting this! |
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further. |
ghost
locked and limited conversation to collaborators
Performing a terraform plan on an existing infrastructure (based on cloudstack and aws providers) that was created using 0.6.6 crashes:
crash.txt
The text was updated successfully, but these errors were encountered: