Terraform to manage GCP based billing alerts

[consideRatio]

It seems that we have created a billing alert in the two-eye-two-see GCP project manually, and that it is getting destroyed by terraform if we make changes to node pools in that project.

Due to that, we are probably required to add billing alerts via terraform and not via the managed cloud console.

Further since changes to node pools coupled to the manually added billing alert, we ran into permissions errors when using terraform that was very hard to understand but could be worked around as documented in #2261. I'm not sure if that PR should or shouldn't be accepted.

Action points

• Figure out what to in more detail

[consideRatio]

We have now used terraform in the two-eye-two-see project's pilot-hubs cluster, and that destroyed the following resources.

  # google_billing_budget.budget[0] will be destroyed
  # (because google_billing_budget.budget is not in configuration)
  - resource "google_billing_budget" "budget" {
      - billing_account = "0157F7-E3EA8C-25AC3C" -> null
      - display_name    = "Billing alert" -> null
      - id              = "billingAccounts/0157F7-E3EA8C-25AC3C/budgets/1bf2106a-0b81-4a70-ab09-bcd568d80e13" -> null
      - name            = "1bf2106a-0b81-4a70-ab09-bcd568d80e13" -> null

      - all_updates_rule {
          - disable_default_iam_recipients   = true -> null
          - monitoring_notification_channels = [
              - "projects/two-eye-two-see/notificationChannels/10693994760737431897",
            ] -> null
          - schema_version                   = "1.0" -> null
        }

      - amount {
          - last_period_amount = false -> null

          - specified_amount {
              - currency_code = "USD" -> null
              - nanos         = 0 -> null
              - units         = "2500" -> null
            }
        }

      - budget_filter {
          - calendar_period        = "MONTH" -> null
          - credit_types           = [] -> null
          - credit_types_treatment = "INCLUDE_ALL_CREDITS" -> null
          - labels                 = {} -> null
          - projects               = [
              - "projects/350668521154",
            ] -> null
          - services               = [] -> null
          - subaccounts            = [] -> null
        }

      - threshold_rules {
          - spend_basis       = "CURRENT_SPEND" -> null
          - threshold_percent = 1 -> null
        }
      - threshold_rules {
          - spend_basis       = "FORECASTED_SPEND" -> null
          - threshold_percent = 1.01 -> null
        }
    }
    
  # google_monitoring_notification_channel.support_email[0] will be destroyed
  # (because google_monitoring_notification_channel.support_email is not in configuration)
  - resource "google_monitoring_notification_channel" "support_email" {
      - display_name = "support@2i2c.org email" -> null
      - enabled      = true -> null
      - force_delete = false -> null
      - id           = "projects/two-eye-two-see/notificationChannels/10693994760737431897" -> null
      - labels       = {
          - "email_address" = "support@2i2c.org"
        } -> null
      - name         = "projects/two-eye-two-see/notificationChannels/10693994760737431897" -> null
      - project      = "two-eye-two-see" -> null
      - type         = "email" -> null
      - user_labels  = {} -> null
    }

[consideRatio]

I think this is outdated now with some terraform logic about it in place.