Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update of node_version for container_cluster with seperately defined node_pool(s) fails #397

Closed
drzero42 opened this issue Sep 8, 2017 · 14 comments · Fixed by GoogleCloudPlatform/magic-modules#1247
Assignees
Labels
bug forward/review In review; remove label to forward service/container

Comments

@drzero42
Copy link
Contributor

drzero42 commented Sep 8, 2017

Terraform Version

Terraform v0.10.4
Terraform-provider-google v0.1.3

Affected Resource(s)

  • google_container_cluster
  • google_container_node_pool

Description

If you have a google_container_cluster with a node pool added through a google_container_node_pool resource and you change the node_version on the google_container_cluster, a 'terraform apply' will start an upgrade, which fails, because it does not specify which node pool it wants to upgrade.

Terraform Configuration Files

https://gist.github.com/drzero42/ebc14c702b2f10572fcd70354110fd03

Steps to Reproduce

  1. Grab manifest from above gist
  2. terraform apply
  3. Wait until both cluster and node_pool resources have been created
  4. Switch node_version from 1.6.9 to 1.7.5 (latest at time of writing)
  5. terraform apply

Error message

Error applying plan:

1 error(s) occurred:

  • google_container_cluster.cluster: 1 error(s) occurred:

  • google_container_cluster.cluster: googleapi: Error 400: Node pool id must be specified if cluster has more than one node pool.., badRequest

@davidquarles
Copy link
Contributor

davidquarles commented Oct 22, 2017

I'm not sure if this merits a new/separate issue or not, but I'd love to see the ability to granularly specify version for each node pool independently. Does that seem reasonable? I can potentially give it a shot later today.

@danawillow
Copy link
Contributor

Hey @davidquarles, that sounds like a separate issue. You're welcome to work on it!

@adam-sandor
Copy link

Any progress on this one? We just ran into the same issue. We have a cluster with multiple node pools and we changed the node version. We are fine with updating all node pools (as long as it's done as a rolling upgrade) but this error is preventing us to do anything through Terraform.

@adam-sandor
Copy link

Update: The workaround is to upgrade all the node pools through gcloud cli or web console.

@sebglon
Copy link
Contributor

sebglon commented Mar 9, 2018

same for me this day with 1.9.2-gke1 and terraform 0.11.3

@danawillow danawillow self-assigned this Mar 27, 2018
@calinah
Copy link

calinah commented May 8, 2018

Hiya is there any progress on this? or a Terraform managed workaround? I'm running terraform v0.11.7 and google provider 1.12.0

@danawillow
Copy link
Contributor

Thanks for the bump! I haven't gotten around to fixing this yet but it's on my list.

@danawillow
Copy link
Contributor

Actually, implementation question for those of you that want this. Right now we do have the ability to update the version on an individual node pool. If you changed the node_version on the cluster with multiple node pools, what would you expect to happen? I can see a few options:

  • It updates all the node pools
  • It updates only the default node pool
  • It gives an error because what you've asked it to do is ambiguous

@calinah
Copy link

calinah commented May 9, 2018

Hi @danawillow I would have thought that since we have the node_version option for the container cluster config and version for the node_pool it would be clear which version refers to the node pool and which to the default pool.

Would it not be a good idea to allow for the update if both flags are specified or update the error message to specify that version flag needs to used on a node_pool to update the default pool?

@tedwardd
Copy link

tedwardd commented May 16, 2018

@danawillow There are two behaviors for this situation that would feel intuitive to me.

The first case would be for node_version to only apply to nodes defined explicitly within the google_container_cluster resource and for version to only apply to the google_container_node_pool resources. That would be the most straight forward behavior to implement in my mind.

If you wanted to get fancy with it I could see node_version only applying to google_container_node_pool resources if version is not also defined within the target pool. But I could see how that might not be possible.

EDIT: Also had another thought. Perhaps node_version should be deprecated and google_container_cluster should follow the same pattern as google_container_node_pool where version is used but placed inside of node_config{} instead of at the top level? This would make it clear what nodes you are affecting where.

@danawillow
Copy link
Contributor

Yeah, I think having it apply to node pools defined in the cluster resource that don't otherwise have a version set (including the default node pool) makes sense. I won't be able to actually get started on this until next week at the earliest, so if anyone else catches this and has an opinion before I start the work, feel free to continue sharing :)

@sebglon
Copy link
Contributor

sebglon commented Jan 10, 2019

I always have this error with:
Terraform v0.11.10

  • provider.google v1.20.0
  • provider.local v1.1.0
  • provider.null v1.0.0
2019-01-10T14:00:32.816+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: 2019/01/10 14:00:32 [DEBUG] Google API Request Details:
2019-01-10T14:00:32.816+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: ---[ REQUEST ]---------------------------------------
2019-01-10T14:00:32.816+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: PUT /v1beta1/projects/akecld-saas-training/locations/europe-west3-a/clusters/europe-west3-a?alt=json&prettyPrint=false HTTP/1.1
2019-01-10T14:00:32.816+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: Host: container.googleapis.com
2019-01-10T14:00:32.816+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: User-Agent: google-api-go-client/0.5 Terraform/0.11.7 (+https://www.terraform.io) terraform-provider-google/1.20.0
2019-01-10T14:00:32.816+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: Content-Length: 49
2019-01-10T14:00:32.816+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: Content-Type: application/json
2019-01-10T14:00:32.816+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: Accept-Encoding: gzip
2019-01-10T14:00:32.816+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:
2019-01-10T14:00:32.816+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: {
2019-01-10T14:00:32.816+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:  "update": {
2019-01-10T14:00:32.816+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:   "desiredNodeVersion": "1.11.6-gke.0"
2019-01-10T14:00:32.816+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:  }
2019-01-10T14:00:32.816+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: }
2019-01-10T14:00:32.816+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:
2019-01-10T14:00:32.816+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: -----------------------------------------------------
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: 2019/01/10 14:00:32 [DEBUG] Google API Response Details:
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: ---[ RESPONSE ]--------------------------------------
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: HTTP/2.0 400 Bad Request
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: Alt-Svc: quic=":443"; ma=2592000; v="44,43,39,35"
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: Cache-Control: private
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: Content-Type: application/json; charset=UTF-8
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: Date: Thu, 10 Jan 2019 13:00:32 GMT
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: Server: ESF
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: Vary: Origin
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: Vary: X-Origin
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: Vary: Referer
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: X-Content-Type-Options: nosniff
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: X-Frame-Options: SAMEORIGIN
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: X-Xss-Protection: 1; mode=block
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: {
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:   "error": {
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:     "code": 400,
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:     "message": "Node_pool_id must be specified.",
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:     "errors": [
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:       {
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:         "message": "Node_pool_id must be specified.",
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:         "domain": "global",
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:         "reason": "badRequest"
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:       }
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:     ],
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:     "status": "INVALID_ARGUMENT"
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:   }
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: }
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: -----------------------------------------------------
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: 2019/01/10 14:00:32 [DEBUG] Unlocking "google-container-cluster/akecld-saas-training/europe-west3-a/europe-west3-a"
2019-01-10T14:00:32.953+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: 2019/01/10 14:00:32 [DEBUG] Unlocked "google-container-cluster/akecld-saas-training/europe-west3-a/europe-west3-a"
2019/01/10 14:00:32 [ERROR] root.project: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:

* google_container_cluster.default: googleapi: Error 400: Node_pool_id must be specified., badRequest
2019/01/10 14:00:32 [ERROR] root.project: eval: *terraform.EvalSequence, err: 1 error(s) occurred:

* google_container_cluster.default: googleapi: Error 400: Node_pool_id must be specified., badRequest
2019/01/10 14:00:33 [DEBUG] plugin: waiting for all plugin processes to complete...

Error: Error applying plan:

1 error(s) occurred:

* module.project.google_container_cluster.default: 1 error(s) occurred:

* google_container_cluster.default: googleapi: Error 400: Node_pool_id must be specified., badRequest

2019-01-10T14:00:33.405+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: 2019/01/10 14:00:33 [ERR] plugin: plugin server: accept unix /tmp/plugin716694891: use of closed network connection
2019-01-10T14:00:33.405+0100 [DEBUG] plugin.terraform: local-exec-provisioner (internal) 2019/01/10 14:00:33 [ERR] plugin: plugin server: accept unix /tmp/plugin170725312: use of closed network connection
2019-01-10T14:00:33.405+0100 [DEBUG] plugin.terraform: local-exec-provisioner (internal) 2019/01/10 14:00:33 [DEBUG] plugin: waiting for all plugin processes to complete...
Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

@sebglon
Copy link
Contributor

sebglon commented Jan 10, 2019

I have tested to remove node_version and add on each node_pool the version directly and now i have another error:

2019/01/10 14:03:07 [DEBUG] Removing 'id' diff and setting Destroy to false because after ignore_changes, this diff no longer requires replacement
2019/01/10 14:03:07 [TRACE] root.project: eval: *terraform.EvalReadDiff
module.project.google_container_cluster.default: Modifying... (ID: europe-west3-a)
  node_pool.0.version: "1.10.7-gke.6" => "1.11.6-gke.0"
  node_pool.1.version: "1.10.7-gke.6" => "1.11.6-gke.0"
2019-01-10T14:03:07.831+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:
panic: interface conversion: interface {} is nil, not string
2019-01-10T14:03:07.831+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:
2019-01-10T14:03:07.831+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: goroutine 367 [running]:
2019-01-10T14:03:07.831+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: github.com/terraform-providers/terraform-provider-google/google.nodePoolUpdate(0xc000416d90, 0x1806920, 0xc0006a06e0, 0xc0007d4d50, 0xc00062e860, 0xc, 0xa, 0x4b, 0x4)
2019-01-10T14:03:07.831+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:       /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-google/google/resource_container_node_pool.go:715 +0x1789
2019-01-10T14:03:07.831+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: github.com/terraform-providers/terraform-provider-google/google.resourceContainerClusterUpdate(0xc000416d90, 0x1806920, 0xc0006a06e0, 0x0, 0xc000abc508)
2019-01-10T14:03:07.831+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:       /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-google/google/resource_container_cluster.go:1199 +0x744
2019-01-10T14:03:07.831+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: github.com/terraform-providers/terraform-provider-google/vendor/github.com/hashicorp/terraform/helper/schema.(*Resource).Apply(0xc0009f0c40, 0xc0006e2050, 0xc000622520, 0x1806920, 0xc0006a06e0, 0x40b801, 0xc0000cdb80, 0x4c11dc)
2019-01-10T14:03:07.831+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:       /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-google/vendor/github.com/hashicorp/terraform/helper/schema/resource.go:231 +0x250
2019-01-10T14:03:07.831+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: github.com/terraform-providers/terraform-provider-google/vendor/github.com/hashicorp/terraform/helper/schema.(*Provider).Apply(0xc00087ff80, 0xc0006e2000, 0xc0006e2050, 0xc000622520, 0xc00005c380, 0x18, 0x7fd22c46e000)
2019-01-10T14:03:07.831+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:       /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-google/vendor/github.com/hashicorp/terraform/helper/schema/provider.go:283 +0x9c
2019-01-10T14:03:07.831+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: github.com/terraform-providers/terraform-provider-google/vendor/github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).Apply(0xc00014fb80, 0xc000622000, 0xc000150420, 0x0, 0x0)
2019-01-10T14:03:07.831+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:       /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-google/vendor/github.com/hashicorp/terraform/plugin/resource_provider.go:527 +0x57
2019-01-10T14:03:07.831+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: reflect.Value.call(0xc0000acae0, 0xc000a25688, 0x13, 0x1b9210e, 0x4, 0xc0000cdf18, 0x3, 0x3, 0xc000541800, 0x0, ...)
2019-01-10T14:03:07.831+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:       /opt/goenv/versions/1.11.0/src/reflect/value.go:447 +0x449
2019-01-10T14:03:07.832+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: reflect.Value.Call(0xc0000acae0, 0xc000a25688, 0x13, 0xc000499f18, 0x3, 0x3, 0x0, 0x0, 0x0)
2019-01-10T14:03:07.832+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:       /opt/goenv/versions/1.11.0/src/reflect/value.go:308 +0xa4
2019-01-10T14:03:07.832+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: net/rpc.(*service).call(0xc000341e40, 0xc0006e3040, 0xc000332e40, 0xc000332e50, 0xc0006d0a00, 0xc000588a60, 0x167a0a0, 0xc000622000, 0x16, 0x167a0e0, ...)
2019-01-10T14:03:07.832+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:       /opt/goenv/versions/1.11.0/src/net/rpc/server.go:384 +0x14e
2019-01-10T14:03:07.832+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4: created by net/rpc.(*Server).ServeCodec
2019-01-10T14:03:07.832+0100 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4:       /opt/goenv/versions/1.11.0/src/net/rpc/server.go:481 +0x47e
2019/01/10 14:03:07 [TRACE] root.project: eval: *terraform.EvalWriteState
2019/01/10 14:03:07 [TRACE] root.project: eval: *terraform.EvalApplyProvisioners
2019/01/10 14:03:07 [TRACE] root.project: eval: *terraform.EvalIf
2019/01/10 14:03:07 [TRACE] root.project: eval: *terraform.EvalWriteState
2019/01/10 14:03:07 [TRACE] root.project: eval: *terraform.EvalWriteDiff
2019/01/10 14:03:07 [TRACE] root.project: eval: *terraform.EvalApplyPost
2019-01-10T14:03:07.833+0100 [DEBUG] plugin: plugin process exited: path=/home/sebglon/git-repo/cloud-customers/saas/projects/saas-training/.terraform/plugins/linux_amd64/terraform-provider-google_v1.20.0_x4
2019/01/10 14:03:07 [ERROR] root.project: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:

* google_container_cluster.default: unexpected EOF
2019/01/10 14:03:07 [ERROR] root.project: eval: *terraform.EvalSequence, err: 1 error(s) occurred:

* google_container_cluster.default: unexpected EOF
2019/01/10 14:03:07 [TRACE] [walkApply] Exiting eval tree: module.project.google_container_cluster.default
2019/01/10 14:03:07 [TRACE] dag/walk: upstream errored, not walking "provisioner.local-exec (close)"
2019/01/10 14:03:07 [TRACE] dag/walk: upstream errored, not walking "meta.count-boundary (count boundary fixup)"
2019/01/10 14:03:07 [TRACE] dag/walk: upstream errored, not walking "module.project.provider.google (close)"
2019/01/10 14:03:07 [TRACE] dag/walk: upstream errored, not walking "root"
2019/01/10 14:03:07 [TRACE] Preserving existing state lineage "f3063d51-91aa-6e52-96e6-46b7cd527f06"
2019/01/10 14:03:07 [TRACE] Preserving existing state lineage "f3063d51-91aa-6e52-96e6-46b7cd527f06"
2019/01/10 14:03:07 [TRACE] Preserving existing state lineage "f3063d51-91aa-6e52-96e6-46b7cd527f06"
2019/01/10 14:03:08 [DEBUG] plugin: waiting for all plugin processes to complete...
2019-01-10T14:03:08.259+0100 [WARN ] plugin: error closing client during Kill: err="connection is shut down"
Error: Error applying plan:

1 error(s) occurred:

* module.project.google_container_cluster.default: 1 error(s) occurred:

* google_container_cluster.default: unexpected EOF

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
2019-01-10T14:03:08.260+0100 [DEBUG] plugin.terraform: local-exec-provisioner (internal) 2019/01/10 14:03:08 [ERR] plugin: plugin server: accept unix /tmp/plugin870984586: use of closed network connection
2019-01-10T14:03:08.260+0100 [DEBUG] plugin.terraform: local-exec-provisioner (internal) 2019/01/10 14:03:08 [DEBUG] plugin: waiting for all plugin processes to complete...
above and apply again to incrementally change your infrastructure.

@ghost
Copy link

ghost commented Feb 18, 2019

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!

@ghost ghost locked and limited conversation to collaborators Feb 18, 2019
@github-actions github-actions bot added service/container forward/review In review; remove label to forward labels Jan 15, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug forward/review In review; remove label to forward service/container
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants