Skip to content
This repository has been archived by the owner on Nov 30, 2018. It is now read-only.

terraform plan fails with unexpected EOF when reading remote state #4

Open
hashibot opened this issue Aug 4, 2017 · 13 comments
Open

terraform plan fails with unexpected EOF when reading remote state #4

hashibot opened this issue Aug 4, 2017 · 13 comments
Labels
bug crash

Comments

@hashibot
Copy link

hashibot commented Aug 4, 2017

This issue was originally opened by @bluedragonx as hashicorp/terraform#15696. It was migrated here as a result of the provider split. The original body of the issue is below.

Terraform Version

Terraform v0.9.11

Terraform Configuration Files

data "terraform_remote_state" "global" {
  backend = "s3"

  config {
    region     = "us-west-2"
    bucket     = "redacted"
    key        = "zo/global.tfstate"
    lock_table = "terraform-state-lock"
  }
}

Debug Output

https://gist.github.com/BlueDragonX/dbce26f86fabb841c26e27b6e66d0e82

Expected Behavior

I expect the terraform plan to run successfully.

Actual Behavior

I get this instead:

Error refreshing state: 2 error(s) occurred:

* data.terraform_remote_state.global: 1 error(s) occurred:

* data.terraform_remote_state.global: data.terraform_remote_state.global: unexpected EOF
* data.terraform_remote_state.net: 1 error(s) occurred:

* data.terraform_remote_state.net: data.terraform_remote_state.net: unexpected EOF

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

  1. Use remote state.
  2. terraform plan

Important Factoids

I've done some digging in the code to see why exactly this is happening. The important bit in the debug log is panic: Unknown: <invalid reflect.Value> which comes from builtin/providers/terraform/flatten.go. Based on the stack trace this implies that one of the output's values was parsed incorrectly. Given that output.Value is an interface and Terraform calls json.Unmarshal to build that, I'm concerned that we're running up against some sort of bug.

Here are the outputs for each of the modules in the state file (mildly redacted):

$ jq '.modules[] | .outputs' global.tfstate
{
  "domain": {
    "sensitive": false,
    "type": "string",
    "value": "redacted."
  },
  "hosted_zone_id": {
    "sensitive": false,
    "type": "string",
    "value": "redacted"
  }
}
{}
{
  "account": {
    "sensitive": false,
    "type": "string",
    "value": "redacted"
  },
  "accounts": {
    "sensitive": false,
    "type": "map",
    "value": {
      "zd": "redacted",
      "zh": "redacted",
      "zi": "redacted",
      "zo": "redacted",
      "zp": "redacted",
      "zs": "redacted"
    }
  },
  "coreos_ami": {
    "sensitive": false,
    "type": "string",
    "value": "redacted"
  },
  "datacenter": {
    "sensitive": false,
    "type": "string",
    "value": "uw2"
  },
  "ecr_account": {
    "sensitive": false,
    "type": "string",
    "value": "redacted"
  },
  "ecr_domain": {
    "sensitive": false,
    "type": "string",
    "value": "redacted"
  },
  "ecr_region": {
    "sensitive": false,
    "type": "string",
    "value": "us-west-2"
  },
  "env": {
    "sensitive": false,
    "type": "string",
    "value": "zo"
  },
  "prefix": {
    "sensitive": false,
    "type": "string",
    "value": "zo-uw2-"
  },
  "region": {
    "sensitive": false,
    "type": "string",
    "value": "us-west-2"
  },
  "zone_prefix": {
    "sensitive": false,
    "type": "map",
    "value": {
      "us-west-2a": "zo-uw2a-",
      "us-west-2b": "zo-uw2b-",
      "us-west-2c": "zo-uw2c-"
    }
  }
}

I'm not sure why that would be parsed by json.Unmarshal incorrectly but I'm going to experiment to see why.

What I really don't get is why this has worked fine for me previously using the same version but is breaking now. I've deployed this same state in other envirnments and it works.
@hashibot hashibot added the crash label Aug 4, 2017
@hashibot hashibot mentioned this issue Aug 4, 2017
Closed
@scream314
Copy link

scream314 commented Aug 4, 2017 •
edited
Loading

TF used: Terraform v0.10.0

After upgrading , I am getting unexpected EOF for all HEREDOCs in aws resources. But if I use HEREDOC with resources provided by (for example) the terraform-provider-libvirt provider, they are accepted.
What really funny is, the number of errors are changing if I run plan again.

In the logs there is first

2017/08/04 17:04:31 [DEBUG] plugin: terraform-provider-aws_v0.1.3_x4: panic: runtime error: invalid memory address or nil pointer dereference
2017/08/04 17:04:31 [DEBUG] plugin: terraform-provider-aws_v0.1.3_x4: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x23a327f]

then a lot of

2017/08/04 17:04:31 [DEBUG] dag/walk: upstream errored, not walking "X.Y.Z.A"

Is the connection to AWS interrupted? That would explain why the number of errors is inconsistent (depends on what is left as it fails).

Ok, I did a run with -parallelism=1, the error is now connection is shut down. With -parallelism=10 I get only unexpected EOFs, with -parallelism=5 a mix of the two.

Update: It is something with the network or the AWS API, because I tried terraform plan from another WAN/ISP link and it worked.

Update2: I do not know what happened/changed, but as I tried today I was able to do terraform plan from anywhere. So my issue is resolved.

Update3: The panic happened again, found out it is unrelated to this issue. My problem seems to be fixed in hashicorp/terraform-provider-aws@3a38399.

@radeksimko radeksimko added the bug label Aug 7, 2017
@mykhailoponomarov
Copy link

Hi all,
I'm experiencing the same error on 0.10, while 0.9.11 works as expected.

@catsby
Copy link
Contributor

catsby commented Aug 9, 2017

Hey @mikhailponomaryov , the previous user reported that hashicorp/terraform-provider-aws#899 has addressed their issue. That patch should be included in the Terraform AWS Provider since version 0.1.2, do you know what version you're on?

@mykhailoponomarov
Copy link

@catsby, Thanks a lot.
I have managed to get it worked with the version constraint set to "0.1.4".

@scream314
Copy link

scream314 commented Aug 10, 2017 •
edited
Loading

@catsby The error happened with terraform-provider-aws_v0.1.3_x4 and was gone as I go geted the AWS Provider and used that version.

@mgresko
Copy link

mgresko commented Nov 7, 2017

I see the same issue going from TF 10.7 -> 10.8. I have tried various version of the aws provider all resulting in the same unexpected EOF. Specifically tried: 0.1.4 and 1.2.0. Stuck at the moment.

@mgresko
Copy link

mgresko commented Nov 7, 2017

017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe: panic: Unknown: <invalid reflect.Value>

This happens on a remote state file. If you run plan directly in the project that created the remote state it works fine. It is just the remote state call.

2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe: panic: Unknown: <invalid reflect.Value>
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe:
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe: goroutine 55 [running]:
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe: github.com/terraform-providers/terraform-provider-terraform/terraform.flatten(0xc0425d4f30, 0xc0423662e0, 0x7, 0x0, 0x0, 0x0)
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe:         /opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-terraform/terraform/flatten.go:48 +0x4f4
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe: github.com/terraform-providers/terraform-provider-terraform/terraform.remoteStateFlatten(0xc0423457b0, 0xc0423457b0)
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe:         /opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-terraform/terraform/flatten.go:21 +0x11e
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe: github.com/terraform-providers/terraform-provider-terraform/terraform.dataSourceRemoteStateRead(0xc0423cc150, 0x0, 0x0, 0xc04238f1e0, 0x16)
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe:         /opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-terraform/terraform/data_source_state.go:114 +0x7cf
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe: github.com/terraform-providers/terraform-provider-terraform/vendor/github.com/hashicorp/terraform/helper/schema.(*Resource).ReadDataApply(0xc0423177a0, 0xc04214d460, 0x0, 0x0, 0xc04233acb8, 0x1, 0x80000000000)
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe:         /opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-terraform/vendor/github.com/hashicorp/terraform/helper/schema/resource.go:259 +0xb7
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe: github.com/terraform-providers/terraform-provider-terraform/vendor/github.com/hashicorp/terraform/helper/schema.(*Provider).ReadDataApply(0xc0422f1e30, 0xc0421485a0, 0xc04214d460, 0x3ab0d50, 0x0, 0x0)
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe:         /opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-terraform/vendor/github.com/hashicorp/terraform/helper/schema/provider.go:426 +0xa1
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe: github.com/terraform-providers/terraform-provider-terraform/vendor/github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).ReadDataApply(0xc04214cee0, 0xc042141cd0, 0xc042141d00, 0x0, 0x0)
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe:         /opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-terraform/vendor/github.com/hashicorp/terraform/plugin/resource_provider.go:604 +0x55
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe: reflect.Value.call(0xc042316480, 0xc042156648, 0x13, 0x22e278d, 0x4, 0xc042387f20, 0x3, 0x3, 0x0, 0x0, ...)
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe:         /usr/local/go/src/reflect/value.go:434 +0x90d
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe: reflect.Value.Call(0xc042316480, 0xc042156648, 0x13, 0xc042387f20, 0x3, 0x3, 0x0, 0x0, 0x0)
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe:         /usr/local/go/src/reflect/value.go:302 +0xab
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe: net/rpc.(*service).call(0xc04215fe40, 0xc04215d310, 0xc0422aa5b8, 0xc042158b00, 0xc0422a4660, 0x1b609c0, 0xc042141cd0, 0x16, 0x1b60a00, 0xc042141d00, ...)
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe:         /usr/local/go/src/net/rpc/server.go:381 +0x149
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe: created by net/rpc.(*Server).ServeCodec
2017-11-07T16:23:12.341-0500 [DEBUG] plugin.terraform-provider-terraform_v1.0.2_x4.exe:         /usr/local/go/src/net/rpc/server.go:475 +0x372
2017/11/07 16:23:12 [ERROR] root: eval: *terraform.EvalReadDataApply, err: data.terraform_remote_state.management: unexpected EOF
2017/11/07 16:23:12 [ERROR] root: eval: *terraform.EvalSequence, err: data.terraform_remote_state.management: unexpected EOF
2017/11/07 16:23:12 [TRACE] [walkRefresh] Exiting eval tree: data.terraform_remote_state.management
2017/11/07 16:23:12 [TRACE] dag/walk: upstream errored, not walking "provider.terraform (close)"

@mgresko
Copy link

mgresko commented Nov 7, 2017

FYI running terraform refresh fixed this issue. Sigh.

@burdiyan
Copy link

I constantly face this issue when updating resources. After getting unexpected EOF, the resource that was refreshing somehow is removed from the state, and then on the next apply is trying to create again, which completely breaks the workflow, because the resource actually exist.

@thiagocaiubi
Copy link

I got unexpected EOF when edited something in Azure portal that Terraform doesn't support.

I tried to add a comma separated list to destination_port_range on azurerm_network_security_rule but it only supports an integer, or a range between o and 65535 or * to indicate any port.

Although the UI accepts comma separated list like 80,443 terraform only implements range support. So I updated the rule directly in the portal just to test my setup and forgot to rollback. So when applied terraform plan I was getting unexpected EOF.

Just deleted the rule and everything started to work again.

@burdiyan
Copy link

burdiyan commented Jan 2, 2018

OMG, I have to take my words back. The EOF issue was because of a panic in our custom provider! I figured it out setting TF_LOG env variable.

@RyanJarv
Copy link

RyanJarv commented Jan 4, 2018 •
edited
Loading

For anyone else running into the error panic: Unknown: <invalid reflect.Value> I somehow had a output in the remote tfstate that was set to a value of null. Removing the output there fixed the issue.

@cregkly
Copy link

cregkly commented Jan 11, 2018

We also get Unexpected EOF from having a output configured in a remote state, the error in debug throws:
panic: Unknown: %!s(float64=2)

The bad output configuration:

output "foo" {
  value = 2
}

Fix was to have the integer quoted as a string like this:

output "foo" {
  value = "2"
}

The state file the incorrect configuration looks like this:

...
                "foo": {
                    "sensitive": false,
                    "type": "string",
                    "value":  2,
                },
...

It should be

...
                "foo": {
                    "sensitive": false,
                    "type": "string",
                    "value":  "2",
                },
...

Note the the output posted to the console when applying the state will be shown without quotes in both cases.

Outputs:

foo = 2

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug crash
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
10 participants
@thiagocaiubi @catsby @radeksimko @burdiyan @mgresko @scream314 @RyanJarv @mykhailoponomarov @cregkly @hashibot