diff --git a/docs/content/_index.md b/docs/content/_index.md index 294e44cb1440..ef60d84e8dc7 100644 --- a/docs/content/_index.md +++ b/docs/content/_index.md @@ -1,12 +1,16 @@ --- title: "Overview" +weight: 20 +aliases: + - /docs/how-to/types-of-resources + - /how-to/types-of-resources + - /get-started/how-magic-modules-works --- - # Magic Modules Magic Modules is a code generator and CI system that's used to develop the Terraform providers -for Google Platform, [`google`](https://github.com/hashicorp/terraform-provider-google) (or TPG) and +for Google Cloud, [`google`](https://github.com/hashicorp/terraform-provider-google) (or TPG) and [`google-beta`](https://github.com/hashicorp/terraform-provider-google-beta) (or TPGB). Magic Modules allows contributors to make changes against a single codebase and develop both @@ -15,13 +19,97 @@ provider versions simultaneously. After sending a pull request against this repo complete output, running presubmit tests, and updating the providers following your change. -## Getting started +## How Magic Modules works + +Magic Modules can be thought of as a source of truth for how to map a Google Cloud API resource +representation to a Terraform resource (or datasource) representation. Magic Modules uses that mapping +(and additional handwritten code where necessary) to generate "downstream" repositories - in particular, +the Terraform providers for Google Cloud: [`google`](https://github.com/hashicorp/terraform-provider-google) +(or TPG) and [`google-beta`](https://github.com/hashicorp/terraform-provider-google-beta) (or TPGB). + +Generation of the downstream repositories happens for every new commit in a PR (to a temporary branch owned by +the [`modular-magician`](https://github.com/modular-magician/) robot user) and on every merge into the main branch +(to the main branch of downstreams). Generation for PR commits allows contributors to manually examine the changes, +as well as allowing automatic running of unit tests, acceptance tests, and automated checks such as breaking change +detection. + +### Resource types + +There are three types of resources supported by Magic Modules: + ++ MMv1 ++ Handwritten ++ DCL/tpgtools + +The following sections describe these tools in detail. + +#### MMv1 + +MMv1 consists of a set of "products"; each product contains one or more "resources". + +Each product has a folder in +[`magic-modules/mmv1/products`](https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/products). +The name of the folder is the "product name", which usually corresponds to the API subdomain covered by the +product (such as `compute.googleapis.com`). Each product folder contains a product configuration file +(`product.yaml`) and one or more resource configuration files (`ResourceName.yaml`). The actual name of a +`ResourceName.yaml` file usually matches the name of a GCP API resource in the product's subdomain. + +MMv1 resource configurations may reference handwritten code stored in +[`magic-modules/mmv1/templates/terraform`](https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/templates/terraform), +which will be injected into the generated resource file. Many MMv1 resources also have one or more handwritten tests, +which are stored in the appropriate service folder inside +[`magic-modules/mmv1/third_party/terraform/services`](https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/third_party/terraform/services). + +In the providers, MMv1-based resources are stored in `PROVIDER/services/PRODUCT/resource_PRODUCT_RESOURCE.go`, where `PROVIDER` +is `google` or `google-beta`, `PRODUCT` is the product name, and RESOURCE is the GCP API resource's name converted to +[snake case ↗](https://en.wikipedia.org/wiki/Snake_case). + +MMv1-based files start with the following header: + +``` +***     AUTO GENERATED CODE    ***    Type: MMv1     *** +``` + +#### Handwritten + +Handwritten resources and datasources are technically part of MMv1; however, they are not generated from YAML configurations. +Instead, they are written as Go code with minimal go template "version guards" to exclude beta-only features from the `google` +provider. + +Handwritten resources and datasources can be grouped by "service", which generally corresponds to the API subdomain the resource +or datasource interacts with. + +In addition to the core implementation, handwritten resources and datasources will also have documentation, tests, and sweepers +(which clean up stray resources left behind by tests). Each type of code is stored in the following locations: + ++ Resource & datasource implementation: In the appropriate service folder inside + [`magic-modules/mmv1/third_party/terraform/services`](https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/third_party/terraform/services) ++ Resource documentation: + [`magic-modules/mmv1/third_party/terraform/website/docs/r`](https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/third_party/terraform/website/docs/r) ++ Datasource documentation: + [`magic-modules/mmv1/third_party/terraform/website/docs/d`](https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/third_party/terraform/website/docs/d) ++ Tests: In the appropriate service folder inside + [`magic-modules/mmv1/third_party/terraform/services`](https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/third_party/terraform/services) ++ Sweepers: [`magic-modules/mmv1/third_party/terraform/utils`](https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/third_party/terraform/utils) + +In the providers, handwritten resources and datasources are stored in `PROVIDER/services/SERVICE/FILENAME.go`, where `PROVIDER` +is `google` or `google-beta`, `SERVICE` is the service name, and `FILENAME` is the name of the handwritten file in magic-modules. +Handwritten files do not have an `AUTO GENERATED CODE` header. + +#### DCL aka tpgtools (maintenance mode) + +DCL / tpgtools is similar to MMv1; however, it is in maintenance mode, which means that new resources using the DCL are not being added. + +DCL-based files start with the following header: + +``` +***     AUTO GENERATED CODE    ***    Type: DCL     *** +``` -Check out the [setup guide]({{< ref "/get-started/generate-providers" >}}) for information on how to set up your environment. ## Other Resources -* [Extending Terraform](https://www.terraform.io/plugin) - * [How Terraform Works](https://www.terraform.io/plugin/how-terraform-works) - * [Writing Custom Providers / Calling APIs with Terraform Providers](https://learn.hashicorp.com/collections/terraform/providers?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS) -* [Terraform Glossary](https://www.terraform.io/docs/glossary) ++ [Extending Terraform](https://www.terraform.io/plugin) + + [How Terraform Works](https://www.terraform.io/plugin/how-terraform-works) + + [Writing Custom Providers / Calling APIs with Terraform Providers](https://learn.hashicorp.com/collections/terraform/providers?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS) ++ [Terraform Glossary](https://www.terraform.io/docs/glossary) diff --git a/docs/content/get-started/contribution-process.md b/docs/content/contribution-process.md similarity index 92% rename from docs/content/get-started/contribution-process.md rename to docs/content/contribution-process.md index bda0300dfbdd..89ff42a68ed0 100644 --- a/docs/content/get-started/contribution-process.md +++ b/docs/content/contribution-process.md @@ -1,14 +1,17 @@ --- title: "Contribution process" -weight: 50 +weight: 9 aliases: - /docs/getting-started/contributing - /getting-started/contributing - /get-started/contributing + - /get-started/contribution-process --- # Contribution process +This page explains how you can contribute code and documentation to the `magic-modules` repository. + ## Before you begin 1. Familiarize yourself with [GitHub flow](https://docs.github.com/en/get-started/quickstart/github-flow) diff --git a/docs/content/develop/breaking-changes/make-a-breaking-change.md b/docs/content/develop/breaking-changes/make-a-breaking-change.md index 47f3e09d227c..bb4e62499955 100644 --- a/docs/content/develop/breaking-changes/make-a-breaking-change.md +++ b/docs/content/develop/breaking-changes/make-a-breaking-change.md @@ -174,7 +174,7 @@ or resource. It is also great to log warnings at runtime if possible. When working on your breaking change, make sure that your base branch is `FEATURE-BRANCH-major-release-{{% param "majorVersion" %}}`. This means that you will follow the standard -[contribution process]({{< ref "/get-started/contribution-process" >}}) +[contribution process]({{< ref "/contribution-process" >}}) with the following changes: 1. Before you start, check out and sync your local `magic-modules` and provider diff --git a/docs/content/develop/custom-code.md b/docs/content/develop/custom-code.md index b769ce1bf83f..c5ae9dea9996 100644 --- a/docs/content/develop/custom-code.md +++ b/docs/content/develop/custom-code.md @@ -5,7 +5,7 @@ weight: 39 # Add custom resource code -This document covers how to add "custom code" to [MMv1 resources]({{< ref "/get-started/how-magic-modules-works#mmv1" >}}). Custom code can be used to add arbitrary logic to a resource while still generating most of the code; it allows for a balance between maintainability and supporting real-worlds APIs that deviate from what MMv1 can support. Custom code should only be added if the desired behavior can't be achieved otherwise. +This document covers how to add "custom code" to [MMv1 resources]({{< ref "/#mmv1" >}}). Custom code can be used to add arbitrary logic to a resource while still generating most of the code; it allows for a balance between maintainability and supporting real-worlds APIs that deviate from what MMv1 can support. Custom code should only be added if the desired behavior can't be achieved otherwise. Most custom code attributes are strings that contain a path to a template file relative to the `mmv1` directory. For example: diff --git a/docs/content/develop/promote-to-ga.md b/docs/content/develop/promote-to-ga.md index 91c64fb3252e..c03bcefcc83f 100644 --- a/docs/content/develop/promote-to-ga.md +++ b/docs/content/develop/promote-to-ga.md @@ -9,7 +9,7 @@ This document describes how to promote an existing resource or field that uses M Handwritten code (including `custom_code`) commonly uses "version guards" in the form of `{{- if ne $.TargetVersionName "ga" }}...{{- end }}` to wrap code that is beta-specific, which need to be removed during promotion. -For more information about types of resources and the generation process overall, see [How Magic Modules works]({{< ref "/get-started/how-magic-modules-works.md" >}}). +For more information about types of resources and the generation process overall, see [How Magic Modules works]({{< ref "/" >}}). ## Before you begin diff --git a/docs/content/develop/resource.md b/docs/content/develop/resource.md index c6ccb54ce843..eaf9dfe454c2 100644 --- a/docs/content/develop/resource.md +++ b/docs/content/develop/resource.md @@ -27,11 +27,11 @@ aliases: This page describes how to add a new resource to the `google` or `google-beta` Terraform provider using MMv1 and/or handwritten code. -For more information about types of resources and the generation process overall, see [How Magic Modules works]({{< ref "/get-started/how-magic-modules-works.md" >}}). +For more information about types of resources and the generation process overall, see [How Magic Modules works]({{< ref "/" >}}). ## Before you begin -1. Complete the [Generate the providers]({{< ref "/get-started/generate-providers" >}}) quickstart to set up your environment and your Google Cloud project. +1. Complete the steps in [Generate the providers]({{< ref "/get-started/generate-providers" >}}) to set up your environment and your Google Cloud project. 2. Ensure that your `magic-modules`, `terraform-provider-google`, and `terraform-provider-google-beta` repositories are up to date. ``` cd ~/magic-modules @@ -46,7 +46,7 @@ For more information about types of resources and the generation process overall {{< tabs "resource" >}} {{< tab "MMv1" >}} -1. Using an editor of your choice, in the appropriate [product folder]({{}}), create a file called `RESOURCE_NAME.yaml`. Replace `RESOURCE_NAME` with the name of the API resource you are adding support for. For example, a configuration file for [NatAddress](https://cloud.google.com/apigee/docs/reference/apis/apigee/rest/v1/organizations.instances.natAddresses) would be called `NatAddress.yaml`. +1. Using an editor of your choice, in the appropriate [product folder]({{}}), create a file called `RESOURCE_NAME.yaml`. Replace `RESOURCE_NAME` with the name of the API resource you are adding support for. For example, a configuration file for [NatAddress](https://cloud.google.com/apigee/docs/reference/apis/apigee/rest/v1/organizations.instances.natAddresses) would be called `NatAddress.yaml`. 2. Copy the following template into the new file: ```yaml # Copyright 2024 Google Inc. diff --git a/docs/content/develop/test/test.md b/docs/content/develop/test/test.md index 4d3bc2c6343b..d6e55784cf1b 100644 --- a/docs/content/develop/test/test.md +++ b/docs/content/develop/test/test.md @@ -26,7 +26,7 @@ For more information about testing, see the [official Terraform documentation](h ## Before you begin -1. Determine whether your resources is using [MMv1 generation or handwritten]({{}}). +1. Determine whether your resources is using [MMv1 generation or handwritten]({{}}). 2. If you are not adding tests to an in-progress PR, ensure that your `magic-modules`, `terraform-provider-google`, and `terraform-provider-google-beta` repositories are up to date. ```bash cd ~/magic-modules diff --git a/docs/content/get-started/generate-providers.md b/docs/content/get-started/generate-providers.md index 41beb7a57c3c..eaf0e1ae109e 100644 --- a/docs/content/get-started/generate-providers.md +++ b/docs/content/get-started/generate-providers.md @@ -192,6 +192,6 @@ gcloud auth revoke ## What's next -- [Learn about Magic Modules]({{< ref "/get-started/how-magic-modules-works.md" >}}) -- [Learn about the contribution process]({{< ref "/get-started/contribution-process.md" >}}) -- [Learn about make commands]({{< ref "/reference/make-commands.md" >}}) +- [Learn about Magic Modules]({{< ref "/" >}}) +- [Learn about the contribution process]({{< ref "/contribution-process.md" >}}) +- [Learn about make commands]({{< ref "/reference/make-commands.md" >}}) \ No newline at end of file diff --git a/docs/content/get-started/how-magic-modules-works.md b/docs/content/get-started/how-magic-modules-works.md deleted file mode 100644 index 520bf739eb70..000000000000 --- a/docs/content/get-started/how-magic-modules-works.md +++ /dev/null @@ -1,59 +0,0 @@ ---- -title: "How Magic Modules works" -weight: 20 -aliases: - - /docs/how-to/types-of-resources - - /how-to/types-of-resources ---- - -# How Magic Modules works - -Magic Modules can be thought of as a source of truth for how to map a GCP API resource representation to a Terraform resource (or datasource) representation. Magic Modules uses that mapping (and additional handwritten code where necessary) to generate "downstream" repositories - in particular, the Terraform providers for Google Cloud: [`google`](https://github.com/hashicorp/terraform-provider-google) (or TPG) and [`google-beta`](https://github.com/hashicorp/terraform-provider-google-beta) (or TPGB). - -Generation of the downstream repositories happens for every new commit in a PR (to a temporary branch owned by the [`modular-magician`](https://github.com/modular-magician/) robot user) and on every merge into the main branch (to the main branch of downstreams). Generation for PR commits allows contributors to manually examine the changes, as well as allowing automatic running of unit tests, acceptance tests, and automated checks such as breaking change detection. - -## Resource types - -There are three types of resources supported by Magic Modules: MMv1, Handwritten, and DCL/tpgtools. These are described in more detail in the following sections. - -### MMv1 - -MMv1 consists of a set of "products"; each product contains one or more "resources". - -Each product has a folder in [`magic-modules/mmv1/products`](https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/products). The name of the folder is the "product name", which usually corresponds to the API subdomain covered by the product (such as `compute.googleapis.com`). Each product folder contains a product configuration file (`product.yaml`) and one or more resource configuration files (`ResourceName.yaml`). The actual name of a `ResourceName.yaml` file usually matches the name of a GCP API resource in the product's subdomain. - -MMv1 resource configurations may reference handwritten code stored in [`magic-modules/mmv1/templates/terraform`](https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/templates/terraform), which will be injected into the generated resource file. Many MMv1 resources also have one or more handwritten tests, which are stored in the appropriate service folder inside [`magic-modules/mmv1/third_party/terraform/services`](https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/third_party/terraform/services) - -In the providers, MMv1-based resources are stored in `PROVIDER/services/PRODUCT/resource_PRODUCT_RESOURCE.go`, where `PROVIDER` is `google` or `google-beta`, `PRODUCT` is the product name, and RESOURCE is the GCP API resource's name converted to [snake case ↗](https://en.wikipedia.org/wiki/Snake_case). - -MMv1-based files start with the following header: - -``` -*** AUTO GENERATED CODE *** Type: MMv1 *** -``` - -### Handwritten - -Handwritten resources and datasources are technically part of MMv1; however, they are not generated from YAML configurations. Instead, they are written as Go code with minimal go template "version guards" to exclude beta-only features from the `google` provider. - -Handwritten resources and datasources can be grouped by "service", which generally corresponds to the API subdomain the resource or datasource interacts with. - -In addition to the core implementation, handwritten resources and datasources will also have documentation, tests, and sweepers (which clean up stray resources left behind by tests). Each type of code is stored in the following locations: - -- Resource & datasource implementation: In the appropriate service folder inside [`magic-modules/mmv1/third_party/terraform/services`](https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/third_party/terraform/services) -- Resource documentation: [`magic-modules/mmv1/third_party/terraform/website/docs/r`](https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/third_party/terraform/website/docs/r) -- Datasource documentation: [`magic-modules/mmv1/third_party/terraform/website/docs/d`](https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/third_party/terraform/website/docs/d) -- Tests: In the appropriate service folder inside [`magic-modules/mmv1/third_party/terraform/services`](https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/third_party/terraform/services) -- Sweepers: [`magic-modules/mmv1/third_party/terraform/utils`](https://github.com/GoogleCloudPlatform/magic-modules/tree/main/mmv1/third_party/terraform/utils) - -In the providers, handwritten resources and datasources are stored in `PROVIDER/services/SERVICE/FILENAME.go`, where `PROVIDER` is `google` or `google-beta`, `SERVICE` is the service name, and `FILENAME` is the name of the handwritten file in magic-modules. Handwritten files do not have an `AUTO GENERATED CODE` header. - -### DCL aka tpgtools (maintenance mode) - -DCL / tpgtools is similar to MMv1; however, it is in maintenance mode, which means that new resources using the DCL are not being added. - -DCL-based files start with the following header: - -``` -*** AUTO GENERATED CODE *** Type: DCL *** -```