Allow overriding the base path for a URL with custom endpoints #1880

rileykarson · 2019-06-04T22:13:00Z

Add support for overriding the base path of the provider. Fixes hashicorp/terraform-provider-google#2962.

This is ready for review. It isn't ready to be merged because I've only added two products for now. They cover what needs to be done for every other product, so once this is approved I'll add the rest and then re-request review.

The test uses the beta endpoint instead of an emulator because it's hard to set up an emulator in an acc test framework. This tests that the feature works, and in the short-medium term we can get this set up to hit GCP staging APIs.

Also, BasePath naming comes from the client libs themselves. CustomEndpoint is a bit closer to what user expectations are. Do you think it's worth changing to use that naming schema? For example, in gcloud it's gcloud config set api_endpoint_overrides/cloudfunctions http://localhost:8008/ to set an endpoint for Cloud Functions.

Equivalent AWS provider feature: https://www.terraform.io/docs/providers/aws/guides/custom-service-endpoints.html

modular-magician · 2019-06-04T22:25:01Z

Hi! I'm the modular magician, I work on Magic Modules.
This PR seems not to have generated downstream PRs before, as of 0f722d9.

Pull request statuses

No diff detected in Ansible.
No diff detected in Inspec.

New Pull Requests

I built this PR into one or more new PRs on other repositories, and when those are closed, this PR will also be merged and closed.
depends: hashicorp/terraform-provider-google-beta#811
depends: GoogleCloudPlatform/terraform-google-conversion#96
depends: hashicorp/terraform-provider-google#3787

modular-magician · 2019-06-04T22:30:48Z

Hi! I'm the modular magician, I work on Magic Modules.
This PR seems not to have generated downstream PRs before, as of 0f722d9.

Pull request statuses

terraform-provider-google-beta already has an open PR.
terraform-google-conversion already has an open PR.
terraform-provider-google already has an open PR.
No diff detected in Ansible.
No diff detected in Inspec.

New Pull Requests

I didn't open any new pull requests because of this PR.

modular-magician · 2019-06-06T15:41:09Z

Hi! I'm the modular magician, I work on Magic Modules.
I see that this PR has already had some downstream PRs generated. Any open downstreams are already updated to your most recent commit, 0f722d9.

Pull request statuses

terraform-provider-google-beta already has an open PR.
terraform-google-conversion already has an open PR.
terraform-provider-google already has an open PR.
No diff detected in Ansible.
No diff detected in Inspec.

New Pull Requests

I didn't open any new pull requests because of this PR.

modular-magician · 2019-06-06T20:46:38Z

Hi! I'm the modular magician, I work on Magic Modules.
I see that this PR has already had some downstream PRs generated. Any open downstreams are already updated to your most recent commit, cd9bd02.

Pull request statuses

terraform-provider-google-beta already has an open PR.
terraform-google-conversion already has an open PR.
terraform-provider-google already has an open PR.
No diff detected in Ansible.
No diff detected in Inspec.

New Pull Requests

I didn't open any new pull requests because of this PR.

modular-magician · 2019-06-07T20:00:29Z

Hi! I'm the modular magician, I work on Magic Modules.
I see that this PR has already had some downstream PRs generated. Any open downstreams are already updated to your most recent commit, 49872b6.

Pull request statuses

terraform-provider-google-beta already has an open PR.
terraform-google-conversion already has an open PR.
terraform-provider-google already has an open PR.
No diff detected in Ansible.
No diff detected in Inspec.

New Pull Requests

I didn't open any new pull requests because of this PR.

modular-magician · 2019-06-07T20:24:19Z

Hi! I'm the modular magician, I work on Magic Modules.
I see that this PR has already had some downstream PRs generated. Any open downstreams are already updated to your most recent commit, 4221f36.

Pull request statuses

terraform-provider-google-beta already has an open PR.
terraform-google-conversion already has an open PR.
terraform-provider-google already has an open PR.
No diff detected in Ansible.
No diff detected in Inspec.

New Pull Requests

I didn't open any new pull requests because of this PR.

modular-magician · 2019-06-07T20:25:28Z

Hi! I'm the modular magician, I work on Magic Modules.
I see that this PR has already had some downstream PRs generated. Any open downstreams are already updated to your most recent commit, bb1f19d.

Pull request statuses

terraform-provider-google-beta already has an open PR.
terraform-google-conversion already has an open PR.
terraform-provider-google already has an open PR.
No diff detected in Ansible.
No diff detected in Inspec.

New Pull Requests

I didn't open any new pull requests because of this PR.

modular-magician · 2019-06-07T20:26:53Z

Hi! I'm the modular magician, I work on Magic Modules.
I see that this PR has already had some downstream PRs generated. Any open downstreams are already updated to your most recent commit, bb1f19d.

Pull request statuses

terraform-provider-google-beta already has an open PR.
terraform-google-conversion already has an open PR.
terraform-provider-google already has an open PR.
No diff detected in Ansible.
No diff detected in Inspec.

New Pull Requests

I didn't open any new pull requests because of this PR.

modular-magician · 2019-06-07T20:46:03Z

Hi! I'm the modular magician, I work on Magic Modules.
I see that this PR has already had some downstream PRs generated. Any open downstreams are already updated to your most recent commit, 76968b9.

Pull request statuses

terraform-provider-google-beta already has an open PR.
terraform-google-conversion already has an open PR.
terraform-provider-google already has an open PR.
No diff detected in Ansible.
No diff detected in Inspec.

New Pull Requests

I didn't open any new pull requests because of this PR.

danawillow

I think it's worth using the endpoint naming schema, but I don't think we need to do it in most of our code- just what we expose to users. Do you think it would be worth having an endpoints block similar to what aws does?

Also, TGC has changes in just forwarding rule and target instance, I assume due to the custom expand changes. Is that ok?

danawillow · 2019-06-07T21:59:49Z

third_party/terraform/utils/provider_test.go.erb

@@ -164,6 +164,37 @@ func TestProvider_loadCredentialsFromJSON(t *testing.T) {
 	}
 }

+func TestAccProviderBasePath_setBasePath(t *testing.T) {


Idea: right now, if the base path stuff completely did nothing, this test would still pass. What if, instead, you set the base path to something fake, and ensured that the resource didn't create?

Sg- adding a negative test.

templates/terraform/resource.erb

rileykarson · 2019-06-07T22:42:51Z

Using a block would be nice, but that prevents us from using environment variables to configure defaults unless the block has been defined. Based on my read of the equivalent gcloud feature (it's not documented), many users internally used it with env vars.

Most of TGC uses an artificial path scheme (it's correct for newer resources, but different than real URIs for older ones), so I didn't touch that. After this change, they'll need to provide a provider config with base paths set to their default values in the validator library that depends on TGC.

modular-magician · 2019-06-07T23:06:07Z

Hi! I'm the modular magician, I work on Magic Modules.
I see that this PR has already had some downstream PRs generated. Any open downstreams are already updated to your most recent commit, 0a9baf0.

Pull request statuses

terraform-provider-google-beta already has an open PR.
terraform-google-conversion already has an open PR.
terraform-provider-google already has an open PR.
No diff detected in Ansible.
No diff detected in Inspec.

New Pull Requests

I didn't open any new pull requests because of this PR.

templates/terraform/resource.erb

modular-magician · 2019-06-10T23:33:36Z

Hi! I'm the modular magician, I work on Magic Modules.
I see that this PR has already had some downstream PRs generated. Any open downstreams are already updated to your most recent commit, b5104fc.

Pull request statuses

terraform-provider-google-beta already has an open PR.
terraform-google-conversion already has an open PR.
terraform-provider-google already has an open PR.
No diff detected in Ansible.
No diff detected in Inspec.

New Pull Requests

I didn't open any new pull requests because of this PR.

modular-magician · 2019-06-11T00:01:06Z

Hi! I'm the modular magician, I work on Magic Modules.
I see that this PR has already had some downstream PRs generated. Any open downstreams are already updated to your most recent commit, 08ab114.

Pull request statuses

terraform-provider-google-beta already has an open PR.
terraform-google-conversion already has an open PR.
terraform-provider-google already has an open PR.
No diff detected in Ansible.
No diff detected in Inspec.

New Pull Requests

I didn't open any new pull requests because of this PR.

modular-magician · 2019-06-11T16:00:17Z

Hi! I'm the modular magician, I work on Magic Modules.
I see that this PR has already had some downstream PRs generated. Any open downstreams are already updated to your most recent commit, c39026a.

Pull request statuses

terraform-provider-google-beta already has an open PR.
terraform-google-conversion already has an open PR.
terraform-provider-google already has an open PR.
No diff detected in Ansible.
No diff detected in Inspec.

New Pull Requests

I didn't open any new pull requests because of this PR.

rileykarson · 2019-06-11T16:04:43Z

@danawillow can you review for the changes between these commits? https://github.com/GoogleCloudPlatform/magic-modules/pull/1880/files/8701c8b78e280cc05d2ca3222202e2a4ba0feb3a..d53ba09c07526bc07d6b460306f811307bb5a327

edit: Looks like the sweeper uses a partial Config object, so I'll need to patch that.

edit again: Actually nope, forgot some other stuff too. I'll add you back when I've fixed it.

modular-magician · 2019-06-11T16:11:45Z

Hi! I'm the modular magician, I work on Magic Modules.
I see that this PR has already had some downstream PRs generated. Any open downstreams are already updated to your most recent commit, d53ba09.

Pull request statuses

terraform-provider-google-beta already has an open PR.
terraform-google-conversion already has an open PR.
terraform-provider-google already has an open PR.
No diff detected in Ansible.
No diff detected in Inspec.

New Pull Requests

I didn't open any new pull requests because of this PR.

modular-magician · 2019-06-11T17:11:42Z

Hi! I'm the modular magician, I work on Magic Modules.
I see that this PR has already had some downstream PRs generated. Any open downstreams are already updated to your most recent commit, b84046f.

Pull request statuses

terraform-provider-google-beta already has an open PR.
terraform-google-conversion already has an open PR.
terraform-provider-google already has an open PR.
No diff detected in Ansible.
No diff detected in Inspec.

New Pull Requests

I didn't open any new pull requests because of this PR.

modular-magician · 2019-06-11T17:44:03Z

Hi! I'm the modular magician, I work on Magic Modules.
I see that this PR has already had some downstream PRs generated. Any open downstreams are already updated to your most recent commit, 4ee0602.

Pull request statuses

terraform-provider-google-beta already has an open PR.
terraform-google-conversion already has an open PR.
terraform-provider-google already has an open PR.
No diff detected in Ansible.
No diff detected in Inspec.

New Pull Requests

I didn't open any new pull requests because of this PR.

modular-magician · 2019-06-11T18:12:56Z

Hi! I'm the modular magician, I work on Magic Modules.
I see that this PR has already had some downstream PRs generated. Any open downstreams are already updated to your most recent commit, b9f5e29.

Pull request statuses

terraform-provider-google-beta already has an open PR.
terraform-google-conversion already has an open PR.
terraform-provider-google already has an open PR.
No diff detected in Ansible.
No diff detected in Inspec.

New Pull Requests

I didn't open any new pull requests because of this PR.

rileykarson · 2019-06-11T18:18:13Z

OK, for real now. https://github.com/GoogleCloudPlatform/magic-modules/pull/1880/files/8701c8b78e280cc05d2ca3222202e2a4ba0feb3a..b9f5e29826e7e69a6cbea1ac56663bd31ca2cc58 are the new commits.

third_party/terraform/utils/config.go.erb

third_party/terraform/utils/provider_handwritten_endpoint.go.erb

third_party/terraform/website/docs/provider_reference.html.markdown

modular-magician · 2019-06-11T22:57:13Z

Hi! I'm the modular magician, I work on Magic Modules.
I see that this PR has already had some downstream PRs generated. Any open downstreams are already updated to your most recent commit, 24909ae.

Pull request statuses

terraform-provider-google-beta already has an open PR.
terraform-google-conversion already has an open PR.
terraform-provider-google already has an open PR.
No diff detected in Ansible.
No diff detected in Inspec.

New Pull Requests

I didn't open any new pull requests because of this PR.

modular-magician · 2019-06-12T18:10:46Z

Hi! I'm the modular magician, I work on Magic Modules.
I see that this PR has already had some downstream PRs generated. Any open downstreams are already updated to your most recent commit, 3c7f5d6.

Pull request statuses

terraform-provider-google-beta already has an open PR.
terraform-google-conversion already has an open PR.
terraform-provider-google already has an open PR.
No diff detected in Ansible.
No diff detected in Inspec.

New Pull Requests

I didn't open any new pull requests because of this PR.

modular-magician · 2019-06-13T15:19:51Z

Hi! I'm the modular magician, I work on Magic Modules.
I see that this PR has already had some downstream PRs generated. Any open downstreams are already updated to your most recent commit, b5d790e.

Pull request statuses

terraform-provider-google-beta already has an open PR.
terraform-google-conversion already has an open PR.
terraform-provider-google already has an open PR.
No diff detected in Ansible.
No diff detected in Inspec.

New Pull Requests

I didn't open any new pull requests because of this PR.

Tracked submodules are build/terraform-beta build/terraform-mapper build/terraform build/ansible build/inspec.

googlebot added the cla: yes label Jun 4, 2019

modular-magician added downstream-generated terraform-google-conversion terraform-provider-google-beta terraform-provider-google labels Jun 4, 2019

rileykarson requested a review from danawillow June 7, 2019 20:42

danawillow reviewed Jun 7, 2019

View reviewed changes

rileykarson force-pushed the base-path-override branch from 76968b9 to 0a9baf0 Compare June 7, 2019 22:55

danawillow approved these changes Jun 8, 2019

View reviewed changes

templates/terraform/resource.erb Show resolved Hide resolved

rileykarson force-pushed the base-path-override branch from 0a9baf0 to b5104fc Compare June 10, 2019 23:22

rileykarson marked this pull request as ready for review June 11, 2019 15:51

rileykarson requested a review from danawillow June 11, 2019 16:02

rileykarson removed the request for review from danawillow June 11, 2019 16:18

rileykarson requested a review from danawillow June 11, 2019 18:17

danawillow approved these changes Jun 11, 2019

View reviewed changes

third_party/terraform/utils/config.go.erb Show resolved Hide resolved

third_party/terraform/utils/provider_handwritten_endpoint.go.erb Outdated Show resolved Hide resolved

third_party/terraform/website/docs/provider_reference.html.markdown Show resolved Hide resolved

rileykarson added 16 commits June 13, 2019 08:01

Allow overriding the base path for a URL with custom endpoints

db0205d

Add negative test

c51155a

Change user-facing docs to refer to custom endpoint

7bd14b4

Add ValidateFunc

f257ec4

Plumb through all the products

a2678d6

Minor fixups

0154749

Add documentation

49e6694

Add modification warnings to code comments

540ea66

Rubocop

3c1660f

Fix sweepers, add missing values to config

7084972

Move default base paths because conversion repo is missing most products

e28f142

Fix SQL

b92491e

docs message, comment inconsistency, alphabetize

58b7066

Fix test failures

fba5935

Add missing Redis assignment

18cc29d

Add dataproc to docs

b5d790e

rileykarson force-pushed the base-path-override branch from ba506ac to b5d790e Compare June 13, 2019 15:07

modular-magician added the automerged label Jun 13, 2019

Update tracked submodules -> HEAD on Thu Jun 13 17:21:09 UTC 2019

edc6f93

Tracked submodules are build/terraform-beta build/terraform-mapper build/terraform build/ansible build/inspec.

modular-magician merged commit caf4024 into GoogleCloudPlatform:master Jun 13, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow overriding the base path for a URL with custom endpoints #1880

Allow overriding the base path for a URL with custom endpoints #1880

rileykarson commented Jun 4, 2019 •

edited

Loading

modular-magician commented Jun 4, 2019

modular-magician commented Jun 4, 2019

modular-magician commented Jun 6, 2019

modular-magician commented Jun 6, 2019

modular-magician commented Jun 7, 2019

modular-magician commented Jun 7, 2019

modular-magician commented Jun 7, 2019

modular-magician commented Jun 7, 2019

modular-magician commented Jun 7, 2019

danawillow left a comment

danawillow Jun 7, 2019

rileykarson Jun 7, 2019

rileykarson commented Jun 7, 2019

modular-magician commented Jun 7, 2019

modular-magician commented Jun 10, 2019

modular-magician commented Jun 11, 2019

modular-magician commented Jun 11, 2019

rileykarson commented Jun 11, 2019 •

edited

Loading

modular-magician commented Jun 11, 2019

modular-magician commented Jun 11, 2019

modular-magician commented Jun 11, 2019

modular-magician commented Jun 11, 2019

rileykarson commented Jun 11, 2019

modular-magician commented Jun 11, 2019

modular-magician commented Jun 12, 2019

modular-magician commented Jun 13, 2019

Allow overriding the base path for a URL with custom endpoints #1880

Allow overriding the base path for a URL with custom endpoints #1880

Conversation

rileykarson commented Jun 4, 2019 • edited Loading

modular-magician commented Jun 4, 2019

Pull request statuses

New Pull Requests

modular-magician commented Jun 4, 2019

Pull request statuses

New Pull Requests

modular-magician commented Jun 6, 2019

Pull request statuses

New Pull Requests

modular-magician commented Jun 6, 2019

Pull request statuses

New Pull Requests

modular-magician commented Jun 7, 2019

Pull request statuses

New Pull Requests

modular-magician commented Jun 7, 2019

Pull request statuses

New Pull Requests

modular-magician commented Jun 7, 2019

Pull request statuses

New Pull Requests

modular-magician commented Jun 7, 2019

Pull request statuses

New Pull Requests

modular-magician commented Jun 7, 2019

Pull request statuses

New Pull Requests

danawillow left a comment

Choose a reason for hiding this comment

danawillow Jun 7, 2019

Choose a reason for hiding this comment

rileykarson Jun 7, 2019

Choose a reason for hiding this comment

rileykarson commented Jun 7, 2019

modular-magician commented Jun 7, 2019

Pull request statuses

New Pull Requests

modular-magician commented Jun 10, 2019

Pull request statuses

New Pull Requests

modular-magician commented Jun 11, 2019

Pull request statuses

New Pull Requests

modular-magician commented Jun 11, 2019

Pull request statuses

New Pull Requests

rileykarson commented Jun 11, 2019 • edited Loading

modular-magician commented Jun 11, 2019

Pull request statuses

New Pull Requests

modular-magician commented Jun 11, 2019

Pull request statuses

New Pull Requests

modular-magician commented Jun 11, 2019

Pull request statuses

New Pull Requests

modular-magician commented Jun 11, 2019

Pull request statuses

New Pull Requests

rileykarson commented Jun 11, 2019

modular-magician commented Jun 11, 2019

Pull request statuses

New Pull Requests

modular-magician commented Jun 12, 2019

Pull request statuses

New Pull Requests

modular-magician commented Jun 13, 2019

Pull request statuses

New Pull Requests

rileykarson commented Jun 4, 2019 •

edited

Loading

rileykarson commented Jun 11, 2019 •

edited

Loading