Bug 1759617: pkg/terraform/exec/plugins: Bump Terraform AWS to 2.36.0 #2676

wking · 2019-11-15T06:07:35Z

Also bump the AWS SDK to the latest 1.25.35. Changelogs here and here. Among other things, this brings in a fix for the eventual-consistency race:

level=error msg="Error: \"terraform-20191008145747927600000001\": NoSuchBucket: The specified bucket does not exist"
level=error msg="\tstatus code: 404, request id: BCC3822AF0933B5C, host id: QVhV5kSr/ztRk3204PNQoVKN6jT7LY0Pk2vJHEKBdG5W4Eok9T+JmZXj8M/1XtBDP5fYEC4HcOI="
level=error
level=error msg="  on ../tmp/openshift-install-302978166/bootstrap/main.tf line 1, in resource \"aws_s3_bucket\" \"ignition\":"
level=error msg="   1: resource \"aws_s3_bucket\" \"ignition\" {"

Generated with:

$ cd pkg/terraform/exec/plugins
$ emacs Gopkg.toml  # terraform-provider-aws -> =2.36.0
$ dep ensure
Solving failure: No versions of github.com/aws/aws-sdk-go met constraints:
  v1.19.26: Could not introduce github.com/aws/aws-sdk-go@v1.19.26 due to multiple problematic subpackages:
  Subpackage github.com/aws/aws-sdk-go/service/applicationinsights is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/forecastservice is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/iotevents is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/lakeformation is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/personalize is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/qldb is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/servicequotas is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)
  ...
$ emacs Gopkg.toml  # aws-sdk-go -> =1.25.35, the latest per https://github.com/aws/aws-sdk-go/tags https://github.com/aws/aws-sdk-go/blob/v1.25.35/CHANGELOG.md
$ dep ensure

openshift-ci-robot · 2019-11-15T06:07:39Z

@wking: This pull request references Bugzilla bug 1759617, which is invalid:

expected the bug to target the "4.3.0" release, but it targets "4.4.0" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1759617: pkg/terraform/exec/plugins: Bump Terraform AWS to 2.36.0

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Also bump the AWS SDK to the latest 1.25.35. Changelogs in [1,2]. Among other things, this brings in a fix for the eventual-consistency race [3,4]: level=error msg="Error: \"terraform-20191008145747927600000001\": NoSuchBucket: The specified bucket does not exist" level=error msg="\tstatus code: 404, request id: BCC3822AF0933B5C, host id: QVhV5kSr/ztRk3204PNQoVKN6jT7LY0Pk2vJHEKBdG5W4Eok9T+JmZXj8M/1XtBDP5fYEC4HcOI=" level=error level=error msg=" on ../tmp/openshift-install-302978166/bootstrap/main.tf line 1, in resource \"aws_s3_bucket\" \"ignition\":" level=error msg=" 1: resource \"aws_s3_bucket\" \"ignition\" {" Generated with: $ cd pkg/terraform/exec/plugins $ emacs Gopkg.toml # terraform-provider-aws -> =2.36.0 $ dep ensure Solving failure: No versions of github.com/aws/aws-sdk-go met constraints: v1.19.26: Could not introduce github.com/aws/aws-sdk-go@v1.19.26 due to multiple problematic subpackages: Subpackage github.com/aws/aws-sdk-go/service/applicationinsights is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/forecastservice is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/iotevents is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/lakeformation is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/personalize is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/qldb is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.)Subpackage github.com/aws/aws-sdk-go/service/servicequotas is missing. (Package is required by github.com/terraform-providers/terraform-provider-aws@v2.36.0.) ... $ emacs Gopkg.toml # aws-sdk-go -> =1.25.35, the latest per https://github.com/aws/aws-sdk-go/tags https://github.com/aws/aws-sdk-go/blob/v1.25.35/CHANGELOG.md $ dep ensure using: $ dep version dep: version : v0.5.1 build date : 2019-03-20 git hash : faa61893 go version : go1.10.3 go compiler : gc platform : linux/amd64 features : ImportDuringSolve=false [1]: https://github.com/terraform-providers/terraform-provider-aws/blob/v2.36.0/CHANGELOG.md#2360-november-14-2019 [2]: https://github.com/aws/aws-sdk-go/blob/v1.25.35/CHANGELOG.md [3]: https://bugzilla.redhat.com/show_bug.cgi?id=1759617#c10 [4]: hashicorp/terraform-provider-aws#10863 (comment)

openshift-ci-robot · 2019-11-15T06:08:20Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

wking · 2019-11-15T06:09:09Z

/bugzilla refresh

openshift-ci-robot · 2019-11-15T06:09:14Z

@wking: This pull request references Bugzilla bug 1759617, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

wking · 2019-11-15T06:19:42Z

Ranking in AWS-flake land over the past 48h:

$ curl -s 'https://search.svc.ci.openshift.org/search?name=aws&search=level.error+msg.*in+resource&context=0&maxAge=48h' | jq -r '. | to_entries[].value | to_entries[].value[].context[]' | sed -n 's/.*in resource //p' | sort | uniq -c | sort -n
      2 \"aws_ami_copy\" \"main\":"
      2 \"aws_route53_record\" \"api_internal\":"
      5 \"aws_s3_bucket\" \"ignition\":"
      6 \"aws_route53_record\" \"etcd_a_nodes\":"
      9 \"aws_instance\" \"bootstrap\":"
     13 \"aws_instance\" \"master\":"
$ curl -s 'https://search.svc.ci.openshift.org/search?name=aws&search=level.error+msg.*in+resource.*aws_s3_bucket&context=3&maxAge=48h' | jq -r '. | to_entries[].value | to_entries[].value[].context[]' | sed -n 's/.*Error: //p' | sort | uniq -c | sort -n
      5 NoSuchBucket: The specified bucket does not exist"

wking · 2019-11-15T06:22:59Z

Hmm, from images:

# github.com/openshift/installer/pkg/terraform/exec/plugins/vendor/github.com/terraform-providers/terraform-provider-aws/aws
pkg/terraform/exec/plugins/vendor/github.com/terraform-providers/terraform-provider-aws/aws/awserr.go:18:5: undefined: errors.As

Looks like it needs Go 1.13.

wking · 2019-11-15T06:25:03Z

Looks like everything since Terraform's AWS-provider 2.30.0 needs Go 1.13. Not sure if we can get that bumped at this point, or if it needs to wait for 4.4.

wking · 2019-11-15T06:30:34Z

Go 1.13 bump: openshift/release#5928

wking · 2019-11-15T06:40:50Z

Looking through the provider changelog, there are also a number of "Final retries after timeouts..." fixes, e.g. for instances in 2.28.0 (hashicorp/terraform-provider-aws#9879) that may help with some of our other timeouts. Although over the past 48 hours, all of the instance timeouts have been:

$ curl -s 'https://search.svc.ci.openshift.org/search?name=aws&search=level.error+msg.*in+resource.*aws_instance&context=3&maxAge=48h' | jq -r '. | to_entries[].value | to_entries[].value[].context[]' | sed -n 's/.*Error: //p' | sort | uniq -c | sort -n
     22 Internal error on launch"

The Terraform-provider bump brought in hashicorp/terraform-provider-aws@864285b1bb (Tagging for VPC Endpoints and VPC Endpoint Services, 2019-05-15, hashicorp/terraform-provider-aws#8674, v2.16.0). Also wire this into the deleteARN tree. VPC endpoint ARNs look like: arn:aws:ec2:{region}:{account}:vpc-endpoint/vpce-123...

sdodson · 2019-11-15T13:36:56Z

/hold
The scope of this change and the requisite bump to Go 1.13 leads me to believe we should wait until master branches for 4.4 then consider a backport after it's soaked for a bit.

LorbusChris · 2019-12-04T07:20:40Z

This looks like a good candidate for folding into the Go Modules migration that I'm working on over at #2745, objections?

openshift-ci-robot · 2019-12-04T07:20:49Z

@wking: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

We need this to bump a Terraform provider [1], which we need to fix an AWS provisioning flake [2]. golang-1.13 became available in CI in 407b8e0 (Add Golang 1.13.1 to CI, 2019-10-01, openshift#5236). Skipping 4.3 [3,4] (which is in code-freeze). [1]: openshift/installer#2676 (comment) [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1759617#c10 [3]: openshift#5928 (comment) [4]: openshift#5928 (comment)

sdodson · 2019-12-19T21:29:40Z

/hold cancel
Master is of course open for 4.4, so have at it.

openshift-ci-robot · 2020-01-31T01:34:10Z

@wking: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-aws	`2390bed`	link	`/test e2e-aws`
ci/prow/images	`2390bed`	link	`/test images`
ci/prow/e2e-libvirt	`2390bed`	link	`/test e2e-libvirt`
ci/prow/e2e-openstack	`2390bed`	link	`/test e2e-openstack`
ci/prow/unit	`2390bed`	link	`/test unit`
ci/prow/e2e-aws-scaleup-rhel7	`2390bed`	link	`/test e2e-aws-scaleup-rhel7`
ci/prow/e2e-aws-upgrade	`2390bed`	link	`/test e2e-aws-upgrade`
ci/prow/e2e-aws-fips	`2390bed`	link	`/test e2e-aws-fips`
ci/prow/govet	`2390bed`	link	`/test govet`
ci/prow/tf-lint	`2390bed`	link	`/test tf-lint`
ci/prow/shellcheck	`2390bed`	link	`/test shellcheck`
ci/prow/yaml-lint	`2390bed`	link	`/test yaml-lint`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

sdodson · 2020-02-04T19:50:41Z

/close
This will be part of #2745

openshift-ci-robot · 2020-02-04T19:50:44Z

@sdodson: Closed this PR.

In response to this:

/close
This will be part of #2745

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Nov 15, 2019

openshift-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Nov 15, 2019

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 15, 2019

openshift-ci-robot requested review from jcpowermac and jhixson74 November 15, 2019 06:08

wking force-pushed the bump-terraform-aws branch from c90d808 to 9bba2bd Compare November 15, 2019 06:08

openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Nov 15, 2019

openshift-ci-robot removed the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Nov 15, 2019

wking mentioned this pull request Nov 15, 2019

ci-operator/config/openshift/installer: Bump to golang-1.13 openshift/release#5928

Merged

wking mentioned this pull request Nov 15, 2019

Bug 1738354: pkg/asset/machines/aws/machines: Request encrypted root volumes #2160

Closed

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 15, 2019

openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 4, 2019

LorbusChris mentioned this pull request Dec 4, 2019

Go Modules Migration #2745

Merged

8 tasks

patrickdillon mentioned this pull request Dec 19, 2019

Add Support for Bahrain me-south-1 AWS region #2826

Merged

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 19, 2019

openshift-ci-robot closed this Feb 4, 2020

wking deleted the bump-terraform-aws branch February 5, 2020 01:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug 1759617: pkg/terraform/exec/plugins: Bump Terraform AWS to 2.36.0 #2676

Bug 1759617: pkg/terraform/exec/plugins: Bump Terraform AWS to 2.36.0 #2676

wking commented Nov 15, 2019

openshift-ci-robot commented Nov 15, 2019

openshift-ci-robot commented Nov 15, 2019

wking commented Nov 15, 2019

openshift-ci-robot commented Nov 15, 2019

wking commented Nov 15, 2019

wking commented Nov 15, 2019

wking commented Nov 15, 2019

wking commented Nov 15, 2019

wking commented Nov 15, 2019

sdodson commented Nov 15, 2019

LorbusChris commented Dec 4, 2019

openshift-ci-robot commented Dec 4, 2019

sdodson commented Dec 19, 2019

openshift-ci-robot commented Jan 31, 2020

sdodson commented Feb 4, 2020

openshift-ci-robot commented Feb 4, 2020

Bug 1759617: pkg/terraform/exec/plugins: Bump Terraform AWS to 2.36.0 #2676

Bug 1759617: pkg/terraform/exec/plugins: Bump Terraform AWS to 2.36.0 #2676

Conversation

wking commented Nov 15, 2019

openshift-ci-robot commented Nov 15, 2019

openshift-ci-robot commented Nov 15, 2019

wking commented Nov 15, 2019

openshift-ci-robot commented Nov 15, 2019

wking commented Nov 15, 2019

wking commented Nov 15, 2019

wking commented Nov 15, 2019

wking commented Nov 15, 2019

wking commented Nov 15, 2019

sdodson commented Nov 15, 2019

LorbusChris commented Dec 4, 2019

openshift-ci-robot commented Dec 4, 2019

sdodson commented Dec 19, 2019

openshift-ci-robot commented Jan 31, 2020

sdodson commented Feb 4, 2020

openshift-ci-robot commented Feb 4, 2020