Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bump terraform providers #83279

Merged
merged 3 commits into from
Apr 2, 2020
Merged

bump terraform providers #83279

merged 3 commits into from
Apr 2, 2020

Conversation

flokli
Copy link
Contributor

@flokli flokli commented Mar 24, 2020

Motivation for this change

This updates the terraform provider update script (thanks @zimbatm for helping out here), and updates all providers.

I haven't finished rebuilding all providers yet.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@flokli flokli requested review from zimbatm, kalbasit and alexfmpe March 24, 2020 11:50
@flokli flokli added the 2.status: work-in-progress This PR isn't done label Mar 24, 2020
@flokli
Copy link
Contributor Author

flokli commented Mar 24, 2020

terraform-providers.google and terraform-providers.google-beta fail to build:

❯ nix-build -A terraform-providers.google
these derivations will be built:
  /nix/store/ml84kwg898dfy9909cpnz5h428651d8q-terraform-provider-google-3.14.0.drv
building '/nix/store/ml84kwg898dfy9909cpnz5h428651d8q-terraform-provider-google-3.14.0.drv'...
unpacking sources
unpacking source archive /nix/store/jjxmsxl0haxj77xf2jd78cxwpaqp211i-source
source root is source
patching sources
configuring
building
go/src/github.com/terraform-providers/terraform-provider-google/google/config.go:51:2: code in directory /build/go/src/github.com/terraform-providers/terraform-provider-google/vendor/google.golang.org/api/sqladmin/v1beta4 expects import "google.golang.org/api/sql/v1beta4"
builder for '/nix/store/ml84kwg898dfy9909cpnz5h428651d8q-terraform-provider-google-3.14.0.drv' failed with exit code 1
error: build of '/nix/store/ml84kwg898dfy9909cpnz5h428651d8q-terraform-provider-google-3.14.0.drv' failed
❯ nix-build -A terraform-providers.google-beta
these derivations will be built:
  /nix/store/1l01hi54y4glw11hi9yb756ijkx4j8v0-terraform-provider-google-beta-3.14.0.drv
building '/nix/store/1l01hi54y4glw11hi9yb756ijkx4j8v0-terraform-provider-google-beta-3.14.0.drv'...
unpacking sources
unpacking source archive /nix/store/dij55sqlwxrpr98ak99w6vnw6m373qif-source
source root is source
patching sources
configuring
building
go/src/github.com/terraform-providers/terraform-provider-google-beta/google-beta/config.go:52:2: code in directory /build/go/src/github.com/terraform-providers/terraform-provider-google-beta/vendor/google.golang.org/api/sqladmin/v1beta4 expects import "google.golang.org/api/sql/v1beta4"
builder for '/nix/store/1l01hi54y4glw11hi9yb756ijkx4j8v0-terraform-provider-google-beta-3.14.0.drv' failed with exit code 1
error: build of '/nix/store/1l01hi54y4glw11hi9yb756ijkx4j8v0-terraform-provider-google-beta-3.14.0.drv' failed

There seems to be something wrong with the vendored libraries I guess?

@flokli
Copy link
Contributor Author

flokli commented Mar 24, 2020

When checking out and building the provider manually, there seems to be some differing behaviour:

$ GO111MODULE=off go build
google/config.go:51:2: code in directory /home/zimbatm/go/src/github.com/terraform-providers/terraform-provider-google/vendor/google.golang.org/api/sqladmin/v1beta4 expects import "google.golang.org/api/sql/v1beta4"

(fails)

$ GO111MODULE=on go build -mod=vendor

(succeeds)

@flokli flokli force-pushed the bump-terraform-providers branch from 05f599b to beb2c00 Compare March 24, 2020 19:09
@flokli
Copy link
Contributor Author

flokli commented Mar 26, 2020

I opened a support ticket at GitHub about the fact that some repositories are missing from the API:

to reproduce, look for terraform-provider-{dns,external,template} in

hub api --paginate https://api.github.com/orgs/terraform-providers/repos | jq | vim -

Or if you want to go fancy, GraphQL:

$ hub api --paginate graphql -f query='
  query($endCursor: String) {
    repositoryOwner(login: "terraform-providers") {
      repositories(first: 100, after: $endCursor) {
        nodes {
          nameWithOwner
          name
          refs(first: 1, refPrefix: "refs/tags/", orderBy: {field: TAG_COMMIT_DATE, direction: DESC}) {
            nodes {
              name                                                                                                                                                                                                                                 
            }                                                                                                                                                                                                                                      
          }                                                                                                                                                                                                                                        
        }                                                                                                                                                                                                                                          
        pageInfo {                                                                                                                                                                                                                                 
          hasNextPage                                                                                                                                                                                                                              
          endCursor                                                                                                                                                                                                                                
        }                                                                                                                                                                                                                                          
      }                                                                                                                                                                                                                                            
    }                                                                                                                                                                                                                                              
  }                                                                                                                                                                                                                                                
' | jq | vim -

@flokli
Copy link
Contributor Author

flokli commented Mar 26, 2020

Alright, already heard back. That was quick! :-)

Turns out, these repos have been moved to the hashicorp organization, and we were fooled by the redirect - so we might want to look at the hashicorp organization too, and filter for terraform-provider-* repos to include these…

@flokli flokli force-pushed the bump-terraform-providers branch from beb2c00 to 5ef47a2 Compare March 30, 2020 10:48
@flokli
Copy link
Contributor Author

flokli commented Mar 30, 2020

I pushed a new version, that uses hub, the GraphQL endpoint, and fetches from both hashicorp and terraform-providers.

This required some refactoring, but now we should have back all terraform providers and not run into GitHubs ratelimiting, as we're not generating a HTTP request for each provider.

PTAL.

@flokli flokli force-pushed the bump-terraform-providers branch 2 times, most recently from 6a1550f to 3123e88 Compare March 30, 2020 10:53
@flokli flokli force-pushed the bump-terraform-providers branch 4 times, most recently from 42c7d97 to af94137 Compare March 30, 2020 11:19
@flokli flokli force-pushed the bump-terraform-providers branch from af94137 to bdbb2a7 Compare March 31, 2020 14:00
@flokli
Copy link
Contributor Author

flokli commented Mar 31, 2020

I bumped the terraform providers once more, now including the fix for hashicorp/terraform-provider-google#5970.

@flokli flokli requested a review from zimbatm March 31, 2020 14:02
@flokli flokli force-pushed the bump-terraform-providers branch from bdbb2a7 to c683e86 Compare April 2, 2020 12:19
@flokli
Copy link
Contributor Author

flokli commented Apr 2, 2020 •
edited
Loading

Some providers that were moved to the hashicorp organization still have references to the terraform-providers organization, which needs patching. Working on this now.

flokli and others added 2 commits April 2, 2020 17:53
@flokli @zimbatm
terraform-providers: refactor
2e19bc7 
Some terraform providers were moved to the `hashicorp` organization,
some are still in the `terraform-providers` organization.

GitHub doesn't show moved repositories when querying via the API (only
on the web interface). In addition, if there are more than 100 results
(there still are), we'd need to implement paging, which is cubersome in curl.

In addition to that, to look up tags, we used to creating a HTTP request
for each and every provider, which caused us to easily get blocked by
GitHubs rate-limiting.

Instead of all that, let's use `hub` (which implements paging) and the
GitHub's GraphQL endpoint to get a list all repositories for a given
organization AND the latest tag for each of them.

We post-process the output with `jq`.

This also removes the providers.txt and folds the data into the script.
The blacklist was becoming too annoying to handle as a config file.

For the whitelist, we still need to individually fetch tags.

Fixes NixOS#83215

Co-Authored-By: zimbatm <zimbatm@zimbatm.com>
@zimbatm @flokli
terraform-providers: fix the google and google-beta providers
3e5149a
@flokli flokli force-pushed the bump-terraform-providers branch from c683e86 to 3090ef7 Compare April 2, 2020 16:28
@flokli
Copy link
Contributor Author

flokli commented Apr 2, 2020 •
edited
Loading

I sent PRs to all of the providers that were renamed:

and added workarounds in special-providers for the time being.

I disabled the broken terraform providers

  • terraform-providers/terraform-provider-circonus
  • terraform-providers/terraform-provider-vmc

Apart from that, I could build terraform-full and terraform_0_11-full.

@flokli flokli force-pushed the bump-terraform-providers branch from 3090ef7 to 9f270c6 Compare April 2, 2020 16:33
@flokli
Copy link
Contributor Author

flokli commented Apr 2, 2020

@GrahamcOfBorg build terraform-full terraform_0_11-full

@flokli flokli requested a review from edef1c April 2, 2020 16:35
@flokli flokli mentioned this pull request Apr 2, 2020
Merged
10 tasks
@flokli flokli merged commit 26afaba into NixOS:master Apr 2, 2020
@flokli flokli deleted the bump-terraform-providers branch April 2, 2020 17:00
This was referenced Jul 6, 2020
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zimbatm zimbatm zimbatm approved these changes

@kalbasit kalbasit Awaiting requested review from kalbasit

@alexfmpe alexfmpe Awaiting requested review from alexfmpe

@edef1c edef1c Awaiting requested review from edef1c

Assignees
No one assigned
Labels
2.status: work-in-progress This PR isn't done 8.has: clean-up 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@flokli @zimbatm