stages: gke-multitenant complains with IP_SPACE_EXHAUSTED error

[gustavovalverde]

Deploying the 3-gke-multitenant stage with this configuration:

clusters = {
  cluster-0 = {
    location = "us-east1"
    vpc_config = {
      subnetwork = "https://www.googleapis.com/compute/v1/projects/XXX-dev-net-spoke-0/regions/us-east1/subnetworks/dev-default-ue1"
    }
  }
}

nodepools = {
  cluster-0 = {
    nodepool-0 = {
      max_pods_per_node = 10
      node_config = {
        disk_type    = "pd-ssd"
        machine_type = "c2-standard-16"
        spot         = true
      }
    }
  }
}

fleet_features = null

fleet_workload_identity = false

This is using the default values already available in the 2-networking-d-separate-envs stage:

region: us-east1
ip_cidr_range: 10.127.48.0/24
description: Default subnet for dev
secondary_ip_ranges:
   pods: 100.64.0.0/24
   services: 100.64.1.0/24

[juliocc]

Hi @gustavovalverde

Sounds like you're running out of IPs in your subnet. Do you have other resources using IP addresses in that subnet? What do you see in the IP utilization of Network Analyzer?

Also, can you share the terraform plan and execution log?

[gustavovalverde]

Hi @juliocc ,

By that time the subnet was empty as I had to recreate it to add the pods and services secondary IP ranges.

I managed to workaround this by adding a max_pods_per_node = 50 and making the secondary subnet bigger. Based on the official documentation and the defaults for 2-networking-d-separate-envs (with /24), it just allows a single node, but 3 nodes (HA) gets deployed using the 3-gke-multitentant default values.

[juliocc]

@gustavovalverde I understand you fixed the issue. Can we close this?

Perhaps we should make the default subnet have a larger pod range. A /20 perhaps?

[ludoo]

Closing this, feel free to reopen if you still think we should fix something.

[gustavovalverde]

I think this should be fixed as not a single example from this README can be deployed with the actual pods and services networking CIDR in https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/blob/master/fast/stages/2-networking-d-separate-envs/data/subnets/dev/dev-dataplatform-ew1.yaml

I've fixed this on my side, but this will always impact newcomers even testing the stages with most defaults, as the multitenant-fleet examples deploy at least 3 nodes, which will always exhaust the IP address range.

I'd say that at least a /16 would be needed here for most example to work without further changes.

[danielmahon]

@juliocc ran into this error while running stage 2-networking-a-peering on a test setup for a fresh org:

╷
│ Error: Error creating Subnetwork: googleapi: Error 400: Invalid value for field 'resource.secondaryIpRanges[1].ipCidrRange': '100.64.1.0/24'. Secondary range overlaps with another range in this subnetwork: 100.64.0.0/16., invalid
│
│   with module.dev-spoke-vpc.google_compute_subnetwork.subnetwork["europe-west1/dev-dataplatform-ew1"],
│   on ../../../modules/net-vpc/subnets.tf line 132, in resource "google_compute_subnetwork" "subnetwork":
│  132: resource "google_compute_subnetwork" "subnetwork" {
│
╵

[juliocc]

@juliocc ran into this error while running stage 2-networking-a-peering on a test setup for a fresh org:

╷
│ Error: Error creating Subnetwork: googleapi: Error 400: Invalid value for field 'resource.secondaryIpRanges[1].ipCidrRange': '100.64.1.0/24'. Secondary range overlaps with another range in this subnetwork: 100.64.0.0/16., invalid
│
│   with module.dev-spoke-vpc.google_compute_subnetwork.subnetwork["europe-west1/dev-dataplatform-ew1"],
│   on ../../../modules/net-vpc/subnets.tf line 132, in resource "google_compute_subnetwork" "subnetwork":
│  132: resource "google_compute_subnetwork" "subnetwork" {
│
╵

This is what happens when you submit code without trying it ;)

Let me see what's going on and I'll get back to you.

[juliocc]

@danielmahon we're addressing this with #1697. I'm waiting to hear from a couple of reviewers but in the meantime you can fetch the branch and give it a try.