Creating GCP project in terraform

[stratog]

I'm having issues creating a GCP project from scratch using terraform. The order of operations seems off. I have to rerun apply several times as terraform gets stuck waiting for the project to get created, apis to initialize, networks get get created, and finally after everything else is created, I get permission issues when a GKE cluster even as project owner.

Terraform Version

Terraform v0.9.1

Affected Resource(s)

Please list the resources as a list, for example:

• google_service_account
• google_container_cluster

If this issue appears to affect multiple resources, it may be an issue with Terraform's core, so please mention this.

Terraform Configuration Files

config
tfstate

Debug Output

debug

Expected Behavior

• Projects get created in 1 step, waiting for the dependent resources to become available.
• Terraform waits for Google Services (googleapis) to become ready
• GKE cluster gets created

Actual Behavior

What actually happened?

Errors occur when creating resources that are depending on another resource that terraform is in the process of creating.

google_container_cluster.asia-northeast1-std-m: googleapi: Error 403: Google Compute Engine: Required 'compute.zones.get' permission for 'projects/terraform-test-nonprod/zones/asia-northeast1-a', forbidden

Creating a project from scratch seems to be missing certain permissions. I've done with the google_container_cluster resource with a service account and with local exec with my own credentials. Both are project owners and compute admins among many many other roles that should have access. I have a feeling that because the apis were not ready when it tried to create these resources, it may have partially corrupted the IAM policies.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. Create a GCP project from scratch that mirrors this configuration
2. Errors when creating google_compute_subnetwork, google_service_account, google_dns_managed_zone, google_compute_network, google_compute_firewall, google_container_cluster, and google_project_services. Project not found
3. Project finally fully initializes
4. Errors creating google_dns_managed_zone, google_compute_firewall, google_compute_subnetwork, google_compute_network: not found and cannot be used for API calls., accessNotConfigured
5. Error creating google_container_cluster: not fully initialized with the default service accounts. Please try again later., backendError
6. Networks partially initializes
7. Errors creating google_compute_firewall, google_compute_subnetwork: resource was not found, notFound
8. Network fully initializes
9. Error creating google_container_cluster: Error 403: Google Compute Engine: Required 'compute.zones.get' permission for 'projects/terraform-test-nonprod/zones/asia-northeast1-a', forbidden

[danawillow]

Hey @stratog, thanks for reporting.

I took a look at your config and I think I know what's going on. Let's pull these two resources out:

      "google_project": {
         "terraform-test-nonprod": {
            "billing_account": "****",
            "name": "terraform-test-nonprod",
            "org_id": "****",
            "project_id": "terraform-test-nonprod"
         }
      },
      "google_project_services": {
         "terraform-test-nonprod": {
            "project": "terraform-test-nonprod",
            "services": [
               [...]
            ]
         }
      },

In this section, the project is created using the google_project resource, and APIs are enabled with the google_project_services resource. However, terraform doesn't have any clues at this point as to which one should be created first, or whether they can both be created at the same time. Terraform doesn't create things in the order they're defined in the file, but rather via a dependency graph that it generates. Since neither resource here explicitly references the other, it assumes it can create them in any order. In order to make the dependency explicit, you can have the google_project_services resource depend on the google_project one like this:

      "google_project_services": {
         "terraform-test-nonprod": {
            "project": "${google_project.terraform-test-nonprod.project_id}",
            "services": [
               [...]
            ]
         }
      },

Getting these throughout your config will likely solve a lot of the errors you've been seeing. Try that out and let me know how it goes!

[stratog]

Thanks @danawillow. I found that step and it helped a bit. A challenge I've found around the services, turning on one API will trigger another to be enabled and sometimes we won't have permission to add/disable the api and any step dependent on gets affected.

Also, in another project, I am finding that the compute api services get removed and that caused a lot of issues with creating/deleting resources that we observed above.

Overall, I found it easier to import a project with a few resources created and managed vs creating a new project from scratch.

[danawillow]

Sorry to hear you're still having issues. It sounds like that first one is covered by #13004. Feel free to file issues for any of the other problems you're seeing as well. In the meantime, is there anything else I can help with for this issue or should we call it solved?

[stratog]

Sure, that seems like the next root cause of creating a project from scratch.

[danawillow]

Cool cool. I'm going to go ahead and close this issue then.

[ctavan]

@danawillow I'm seeing similar issues without even managing the google_project itself through terraform.

What I do is to create a new gcloud project manually through the cloud console and then enable all required APIs using google_project_services.

Unfortunately I'm unable to provision my setup in one single terraform run since terraform seems to try to provision some resources whose APIs have not yet been enabled (even though they are listed in the google_project_services.

On the first run I get errors like:

* google_compute_backend_bucket.static: Error creating backend bucket: googleapi: Error 403: Access Not Configured. Compute Engine API has not been used in project *** before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/compute_component/overview?project=*** then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry., accessNotConfigured
* module.environment.module.my_kubernetes_ssl_ingress.google_compute_global_address.main: 1 error(s) occurred:

* google_compute_global_address.main: Error creating address: googleapi: Error 403: Access Not Configured. Compute Engine API has not been used in project *** before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/compute_component/overview?project=*** then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry., accessNotConfigured
* module.environment.module.container_cluster.google_container_cluster.main: 1 error(s) occurred:

* google_container_cluster.main: googleapi: Error 403: Google Compute Engine: Access Not Configured. Compute Engine API has not been used in project *** before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/compute_component/overview?project=*** then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry., forbidden
* module.environment.module.api_kubernetes_ssl_ingress.google_compute_global_address.main: 1 error(s) occurred:

* google_compute_global_address.main: Error creating address: googleapi: Error 403: Access Not Configured. Compute Engine API has not been used in project *** before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/compute_component/overview?project=*** then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry., accessNotConfigured

Repeating terraform apply a few times eventually leads to a "green" setup.

Is there actually any notion of dependency between google_project_services and the other google_* resources?

[danawillow]

Hey @ctavan, I don't think there's anything we can do about that code-wise since the automatic dependency generation relies on referencing a resource from another, which isn't possible in this case. Luckily, there's a configuration parameter that exists for use cases like this: https://www.terraform.io/docs/configuration/resources.html#explicit-dependencies. Adding depends_on should fix that for you. Let me know how that goes!

[ctavan]

Hey @danawillow, thanks for your quick feedback and for the hint about depends_on. I tried that initially but ran into problems since I'm using modules and it seems like depends_on cannot be specified for modules:

    module my_module: depends_on is not a valid parameter

Also for resources there's a problem during the initial run:

* google_storage_bucket.static: resource depends on non-existent resource 'google_project_services.main.project'

So looks like depends_on does not do the trick.

However I think I might have found a different viable workaround: Since most (if not all) of the Google resources accept an optional project parameter I can use google_project_services.XYZ.project to refer to the project. It seems like this creates an implicit dependency and the APIs are enabled before the other Google resources are created. At least I was able to destroy and recreate my environment (which cointains ~40 resources at the moment) with one single terraform apply successfully 3 times in a row…

I was wondering if referring to static, non-computed attributes like google_project_services.XYZ.project in this example actually makes terraform create an implicit dependency? Do you know that for sure (I'm not that familiar with the terraform core codebase yet)? From the console output that I see it appears to be the case since any google-component is now only created after google_project_services reporst Creation complete.

Anyways, here's an example of what appears to work well for me:

resource "google_project_services" "main" {
  project  = "my-project"
  services = [
    "bigquery-json.googleapis.com",
    "cloudapis.googleapis.com",
    "cloudtrace.googleapis.com",
    "compute-component.googleapis.com",
    "container.googleapis.com",
    "dataflow.googleapis.com",
    "logging.googleapis.com",
    "monitoring.googleapis.com",
    "pubsub.googleapis.com",
    "sqladmin.googleapis.com"
  ]
}

resource "google_storage_bucket" "static" {
  name     = "my-bucket-name"
  project  = "${google_project_services.main.project}"
  location = "US"
}

[danawillow]

Oh, great idea! Your intuition is correct- that does create the dependency. If you're ever curious again about the dependency graph that terraform creates, you can try using terraform graph: https://www.terraform.io/docs/commands/graph.html

[ctavan]

Thanks for the hint! I actually even tried that but found it a bit hard to reason about the result ( terraform graph -draw-cycles | dot -Tpdf > graph.pdf ):

Is there any better way to visualize these graphviz-graphs other than converting them to a PDF? (Maybe becomes a bit too off-topic though…)

[danawillow]

Yeah we're a bit past my area of expertise but I'm sure someone on the internet has a good answer for you :)

[tbernacchi]

Hi everyone, I'm facing the same issue I think...here are the output of my "terraform apply"

Error: Error applying plan:

1 error(s) occurred:

• google_compute_instance.teste-gcloud: 1 error(s) occurred:

• google_compute_instance.teste-gcloud: Error loading zone 'southamerica-east1-b': googleapi: Error 403: Required 'compute.zones.get' permission for 'projects/eminent-cache-186621/zones/southamerica-east1-b', forbidden

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

And these are my files:

main.tf:

root@lwayslearning:/home/tadeu/terraform# cat main.tf
provider "google" {
credentials = "${file ("/home/tadeu/terraform/google-cloud-terraform-key.json")}"
project = "eminent-cache-186621"
region = "southamerica-east1"
}

root@lwayslearning:/home/tadeu/terraform# cat instances.tf
resource "google_compute_instance" "teste-gcloud" {
name = "teste-1"
machine_type = "n1-standard-1"
zone = "southamerica-east1-b"

    service_account {
            scopes = ["userinfo-email", "compute-ro", "storage-ro"]
    }

boot_disk {
	device_name = "teste-1"
	auto_delete = "true"
	initialize_params {
  		size = 50
  		image = "debian-9-stretch-v20171213"
	}
}

network_interface {
            network = "default"
            access_config {}
    }

connection {
  	type        = "ssh"
  	user        = "${var.ssh_username}"
  	private_key = "${file("${var.private_key_path}")}"
  	agent       = false
}

}

I dont know what else should I do! :(

Any help will be appreciate!!

Tks!!!

[stratog]

@tadeuuuuu Check to see if you've modified your compute service account:

<your_project_id>-compute@developer.gserviceaccount.com should have Editor role in the IAM section.

I ran into this issue when I set IAM permissions, removing anything that was preset, and replaced it with just the handful of permissions that I had set explicitly in terraform.

[danawillow]

Yup, @tadeuuuuu I would just double check the permissions you have granted to whatever service account you're using for credentials. A good strategy I like for debugging this sort of thing is to try to accomplish whatever it is you're trying to do in gcloud using the same service account- if you get the same error, then you know that terraform isn't the issue.

[SebastianUA]

Hello!

Please help me with the following issue:

Error: Error applying plan:

1 error(s) occurred:

* module.project.google_project.project: 1 error(s) occurred:

* google_project.project: Error creating project project-test-8204d5b5 (project-test): googleapi: Error 403: User is not authorized., forbidden.

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

I have tried to use terraform for GCP's modules, however - It's my first time with this cloud and I can't understand which actions I did wrong. I can't create any resources.....

Thanks.

[rahulkp220]

I am also facing similar issues at the time of applying the plan.
Error loading zone 'us-east1-b': googleapi: Error 403: Required 'compute.zones.get' permission for 'projects/myproject/zones/us-east1-b', forbidden

[danawillow]

Hey @rahulkp220, I'd recommend double-checking the permissions on the account you're using to run Terraform.

[rahulkp220]

@danawillow Thanks, but I have given the permission of Compute Engine Admin to this account. It still doesn't work.

Can you tell me the roles I should give to my terraform service account? I need to be able to do everything using this same account.

[danawillow]

That should be fine. Have you tried using that same service account with gcloud to read compute zones? (https://cloud.google.com/sdk/gcloud/reference/auth/activate-service-account). There's also very little debugging we can really do with just an error message, unfortunately. If you'd like to open a new issue and fill out the full template there, that would be a big help in debugging. But try the gcloud thing first :)

[rahulkp220]

Hmm, so there are two accounts to start with.
One that I used to log into the google web console, create a project and another service account(i.e the one to be used by terraform), and
Other is this terraform service account itself.

Right now, I have my first account sitting inside the gcloud configs.(the one that I used with gcloud init)

cat ~/.config/gcloud/configurations/config_default 
[core]
account = rahul@mycompany.com
project = myproject

But for automation, I am using terraform service account. So, do I need to change that email ID with the one that I am using for terraform ?

[rahulkp220]

@danawillow Ohkay, I did authenticate the account by using,

gcloud auth activate-service-account terraform@mycompany.iam.gserviceaccount.com \
--key-file=account-f26c05168d43.json
Activated service account credentials for: [terraform@mycompany.iam.gserviceaccount.com]

$ cat ~/.config/gcloud/configurations/config_default
[core]
account = terraform@mycompany.iam.gserviceaccount.com
project = searchstax

and tried to list the zones. I fell into the following errors.

gcloud compute zones list
ERROR: (gcloud.compute.zones.list) Some requests did not succeed:
 - Required 'compute.zones.list' permission for 'projects/myproject'

[danawillow]

Right, so that seems like something that you need to work out the correct IAM permissions for. If the account doesn't have permissions in gcloud, then it won't have them in Terraform either.

[johnmehler]

I'm having a similar issue. I have a Terraform file that does nothing except enable APIs that I'm running against a project with no APIs enabled. The credentials.json file is a JSON file from a service account that's a Project Owner. There shouldn't be any race conditions, since the only thing the terraform file does it enable APIs.

This is what happens when I run terraform apply:

google_project_services.project: Creating...
  disable_on_destroy:  "" => "true"
  project:             "" => "jmehler-pipeline"
  services.#:          "" => "13"
  services.1610229196: "" => "bigquery-json.googleapis.com"
  services.1954675454: "" => "serviceusage.googleapis.com"
  services.2117420113: "" => "pubsub.googleapis.com"
  services.2240314979: "" => "compute.googleapis.com"
  services.238136042:  "" => "cloudapis.googleapis.com"
  services.2631575801: "" => "sqladmin.googleapis.com"
  services.323125032:  "" => "cloudtrace.googleapis.com"
  services.3237295688: "" => "monitoring.googleapis.com"
  services.3355193353: "" => "logging.googleapis.com"
  services.3445186629: "" => "sourcerepo.googleapis.com"
  services.3731214611: "" => "compute-component.googleapis.com"
  services.3740470850: "" => "container.googleapis.com"
  services.729731224:  "" => "dataflow.googleapis.com"

Error: Error applying plan:

1 error(s) occurred:

* google_project_services.project: 1 error(s) occurred:

* google_project_services.project: Error creating services: failed to issue request: googleapi: Error 403: The caller does not have permission, forbidden

[johnmehler]

Here's the code:

provider "google" {
	credentials = "${file("${var.credentials}")}"
	project		= "jmehler-pipeline"
	region		= "us-west1"
}

resource "google_project_services" "project" {
  project = "jmehler-pipeline"
  services   = [
    "serviceusage.googleapis.com",
    "sourcerepo.googleapis.com",
    "compute.googleapis.com",
    "bigquery-json.googleapis.com",
    "cloudapis.googleapis.com",
    "cloudtrace.googleapis.com",
    "compute-component.googleapis.com",
    "container.googleapis.com",
    "dataflow.googleapis.com",
    "logging.googleapis.com",
    "monitoring.googleapis.com",
    "pubsub.googleapis.com",
    "sqladmin.googleapis.com"
  ]
}

[todd-dsm]

@danawillow Found another aspect of this issue today, the project name/id; example:

* google_project.project: error creating project foo-00-bar-b0d9ca48 (foo-00-bar): googleapi: Error 400: field [project_id] has issue [project_id contains invalid components], badRequest. If you received a 403 error, make sure you have the `roles/resourcemanager.projectCreator` permission

This is a GCP error AND it errors (somewhat?) incorrectly; here's some testing:

1. IF the project_name is foo-00-bar; then Terraform returns the Google API error to the terminal.

In this case project_id: "foo-00-bar-b0d9ca48" => will properly suggest a project_id issue and throws a red herring in the mix with the possibility of a projectCreator issue.

2. IF the project_name is foo; then Terraform builds the project without issue.

The Google docs give no indication that there are rules for project_name or project_id. And, maybe there are none when creating them in the WebUI; that's reflected in the Identifying projects (image) my-sample-project-ananan.

However, at least in this limited circumstance, there seem to be rules when terraforming. I can't find these rules anywhere.

Hope this helps 😄

[danawillow]

Hi @todd-dsm, this is a pretty old, closed issue. If you have a new bug to report please fill out the full template in the google provider issue tracker: https://github.com/terraform-providers/terraform-provider-google/issues.

[pratibhadeepti]

Hi team, I'm trying to create google host and service project from scratch but whenever I try to do terraform apply more then once it deletes my project created with same name and recreates but then it gives project already exists and google removes that projects after 30 days. Kindly help how can I resolve the problem tried using data source but of no use.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.