Allow arbitrary service quota requests (#97)

* add quota codegen

* use defaults vs actuals

* add generic quota capbility

* quota codegen

* update generation script and add notes about duplicates

.gitignore

@@ -22,10 +22,13 @@ out/
 __pycache__
 *.pyc
 
+# Python virtual environment
+.venv
 
 # Go best practices dictate that libraries should not include the vendor directory
 vendor
 
 # Ignore Terraform lock files, as we want to test the Terraform code in these repos with the latest provider
 # versions.
 .terraform.lock.hcl
+test/.go-version

codegen/quotas/README.md

@@ -0,0 +1,44 @@
+# AWS Service Quotas Generator
+
+This Python script is used to generate Terraform files for managing AWS service quota requests. It interacts with the AWS Service Quotas API and fetches information about the quotas for different services. The script then generates Terraform code based on this information and writes it to (`main.tf` and `variables.tf`) files.
+
+## Gotchas 
+
+- Generating the quotas could be time consuming as the script honors the API limits for the used AWS APIs.
+- Certain services have duplicate quotas - same description but different code. Those are handled by appending the quota code to the input variable name.
+
+## Requirements
+- Python 3.6+
+- Boto3
+- AWS CLI (optional, for configuring AWS credentials)
+
+## Usage
+
+Ensure you have valid AWS credentials to access the service quotas service.
+
+### Install Dependencies
+Install the required Python packages by running:
+
+```
+pip install -r requirements.txt
+```
+
+### Command Line Arguments
+The script accepts the following command line arguments:
+
+- `--region` (optional): Specify the AWS region to query service quotas for. Defaults to `us-east-1`.
+- `--outdir` (optional): Output directory for the resulting terraform files. Defaults to `../../modules/request-quota-increase`.
+
+### Running the Script
+To run the script with default settings (region `us-east-1` and output `../../modules/request-quota-increase`):
+
+```
+python generate_quotas.py
+```
+
+To specify a different region and output file:
+
+```
+python generate_quotas.py --region us-west-2 --outdir "./path/to/your/dir"
+
+```

codegen/quotas/generate_quotas.py

@@ -0,0 +1,141 @@
+import argparse
+import os
+import subprocess
+import time
+
+import boto3
+from templates import (
+    get_variable_name,
+    terraform_locals_template,
+    terraform_main,
+    terraform_variable_template,
+    terraform_vars,
+)
+
+# Parse command-line arguments
+parser = argparse.ArgumentParser(
+    description="Generate a markdown document of all adjustable AWS service quotas."
+)
+parser.add_argument(
+    "--region",
+    default="us-east-1",
+    help="AWS region to query service quotas for. Defaults to us-east-1.",
+)
+parser.add_argument(
+    "--outdir",
+    default="../../modules/request-quota-increase",
+    help='Output directory for the resulting terraform files. Defaults to "../../modules/request-quota-increase".',
+)
+args = parser.parse_args()
+
+# Initialize a boto3 client for Service Quotas in the specified region
+client = boto3.client("service-quotas", region_name=args.region)
+
+
+def list_all_services():
+    """List all AWS services that have quotas."""
+    services = []
+    response = client.list_services()
+    services.extend(response["Services"])
+    while "NextToken" in response:
+        time.sleep(0.3)  # Delay to respect rate limits
+        response = client.list_services(NextToken=response["NextToken"])
+        services.extend(response["Services"])
+    return services
+
+
+def list_quotas_for_service(service_code):
+    """List the quotas for a given service by its service code."""
+    print(f"Fetching quotas for service {service_code}")
+    quotas = []
+    response = client.list_aws_default_service_quotas(ServiceCode=service_code)
+    quotas.extend(response["Quotas"])
+    while "NextToken" in response:
+        time.sleep(0.3)  # Delay to respect rate limits
+        response = client.list_aws_default_service_quotas(
+            ServiceCode=service_code, NextToken=response["NextToken"]
+        )
+        quotas.extend(response["Quotas"])
+    return quotas
+
+
+def generate_terraform(services):
+    """
+    Generate Terraform code for the given AWS services.
+
+    This function iterates over the provided services, fetches the quotas for each service,
+    and generates Terraform code for each adjustable quota. If a quota with the same variable name
+    already exists, it appends the quota code to the quota name to make it unique, and stores the
+    duplicate variable in a separate list.
+
+    Parameters:
+    services (list): A list of AWS services. Each service is a dictionary that contains the service details.
+
+    Returns:
+    tuple: A tuple containing two strings. The first string is the Terraform code for the main.tf file,
+    and the second string is the Terraform code for the variables.tf file.
+
+    Prints:
+    For each duplicate variable, it prints a message in the format "Duplicate Variable: {variable_name}: {quota_code}".
+    """
+    terraform_variables = ""
+    terraform_maps = ""
+    unique_variables = set()
+    duplicate_variables = []
+    for service in services:
+        # Adjust this based on your rate limit analysis and AWS documentation
+        time.sleep(0.3)
+        quotas = list_quotas_for_service(service["ServiceCode"])
+        for quota in quotas:
+            if quota["Adjustable"]:
+                variable_name = get_variable_name(
+                    service["ServiceCode"], quota["QuotaName"]
+                )
+                if variable_name in unique_variables:
+                    duplicate_variables.append(f"{variable_name}: {quota['QuotaCode']}")
+                    quota["QuotaName"] = f"{quota['QuotaName']}_{quota['QuotaCode']}"
+                else:
+                    unique_variables.add(variable_name)
+                terraform_variables += terraform_variable_template(
+                    service["ServiceCode"], quota["QuotaName"], quota["QuotaCode"]
+                )
+                terraform_maps += terraform_locals_template(
+                    service["ServiceCode"], quota["QuotaName"], quota["QuotaCode"]
+                )
+    main_tf = terraform_main(terraform_maps)
+    vars_tf = terraform_vars(terraform_variables)
+    for variable in duplicate_variables:
+        print(f"Duplicate Variable: {variable}")
+
+    return main_tf, vars_tf
+
+
+# Fetch all services
+services = list_all_services()
+
+# Generate the Terraform code
+tf_main, tf_vars = generate_terraform(services)
+
+# Ensure the output directory exists
+output_dir = args.outdir
+if not os.path.exists(output_dir):
+    os.makedirs(output_dir)
+
+# Write the main.tf to the specified output directory
+main_tf_path = os.path.join(output_dir, "main.tf")
+with open(main_tf_path, "w") as file:
+    file.write(tf_main)
+
+# Write the variables.tf to the specified output directory
+variables_tf_path = os.path.join(output_dir, "variables.tf")
+with open(variables_tf_path, "w") as file:
+    file.write(tf_vars)
+
+# Run terraform fmt on both files
+subprocess.run(["terraform", "fmt", main_tf_path], check=True)
+subprocess.run(["terraform", "fmt", variables_tf_path], check=True)
+
+# Print the success message
+print(
+    f"Terraform files have been written to {output_dir} and formatted with terraform fmt"
+)

codegen/quotas/requirements.txt

@@ -0,0 +1 @@
+boto3>=1.20.0,<2.0

codegen/quotas/templates.py

@@ -0,0 +1,69 @@
+import re
+
+
+def get_variable_name(service_code, quota_name):
+    variable_name = f"{service_code}_{quota_name}".lower()
+    return re.sub(r'\W+', '_', variable_name)
+
+def terraform_variable_template(service_code, quota_name, quota_code):
+    variable_name = get_variable_name(service_code, quota_name)
+    return f'''variable "{variable_name}" {{
+  description = "Quota for [{service_code}]: {quota_name} ({quota_code})"
+  type        = number
+  default     = null
+}}\n\n'''
+
+def terraform_locals_template(service_code, quota_name, quota_code):
+    variable_name = get_variable_name(service_code, quota_name)
+    return f'''    {variable_name} = {{
+      quota_code    = "{quota_code}"
+      service_code  = "{service_code}"
+      desired_quota = var.{variable_name}
+    }},\n'''
+
+def terraform_main(all_quotas):
+    return f'''# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+# CONFIGURE SERVICE QUOTAS
+# NOTE: This module is autogenerated. Do not modify it manually.
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+terraform {{
+  required_version = ">= 1.0.0"
+  required_providers {{
+    aws = {{
+      source  = "hashicorp/aws"
+      version = ">= 3.75.1, < 6.0.0"
+    }}
+  }}
+}}
+
+locals {{
+  all_quotas = {{
+{all_quotas}
+  }}
+
+  adjusted_quotas = {{
+    for k, v in local.all_quotas : k => v
+    if v.desired_quota != null
+  }}
+}}
+
+resource "aws_servicequotas_service_quota" "increase_quotas" {{
+  for_each = local.adjusted_quotas
+
+  quota_code   = each.value.quota_code
+  service_code = each.value.service_code
+  value        = each.value.desired_quota
+}}
+
+'''
+
+def terraform_vars(all_vars):
+    return f'''# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+# INPUT VARIABLES FOR SERVICE QUOTAS
+# NOTE: This module is autogenerated. Do not modify it manually.
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+{all_vars}
+
+\n'''

examples/request-quota-increase/main.tf

@@ -6,13 +6,11 @@ provider "aws" {
   region = var.aws_region
 }
 
-module "quota-increase" {
+module "quota_increase" {
   source = "../../modules/request-quota-increase"
 
-  resources_to_increase = {
-    # In this example, to avoid opening a new request every time we run an automated test, we are setting the quotas
-    # to their default values. In the real world, you'd want to set these quotes to higher values.
-    nat_gateway = 5
-    nacl_rules  = 20
-  }
+  # In this example, to avoid opening a new request every time we run an automated test, we are setting the quotas
+  # to their default values. In the real world, you'd want to set these quotes to higher values.
+  vpc_rules_per_network_acl              = 20
+  vpc_nat_gateways_per_availability_zone = 5
 }

examples/request-quota-increase/outputs.tf

@@ -1,3 +1,3 @@
 output "new_quotas" {
-  value = module.quota-increase.new_quotas
+  value = module.quota_increase.new_quotas
 }

modules/request-quota-increase/README.md

@@ -1,16 +1,19 @@
 # Request AWS Quota Increase
 
-This module can be used to request a quota increase for an AWS Resource.
+This module can be used to request a quota increase for AWS Resources. The module is [generated](../../codegen/quotas/) using [AWS Service Quotas API](https://docs.aws.amazon.com/servicequotas/2019-06-24/apireference/Welcome.html), and inputs for each adjustable quota for different services are added to the module.
+
+**NOTE:** The service quotas for certain services have duplicate items. Those duplicate quotas have been named differently in the [input variables](./variables.tf) by appending the service quota code at the end of the variable name, e.g. `networkmonitor_number_of_probes_per_monitor` and `networkmonitor_number_of_probes_per_monitor_l_f192a8d6`.
 
 ## Features
 
-- Request a quota increase for Network ACL Rules and NAT Gateway.
+- Request a quota increase for any AWS resource.
 
 ## Learn
 
 ### Core Concepts
 
 - [AWS Service Quotas Documentation](https://docs.aws.amazon.com/servicequotas/?id=docs_gateway)
+- [AWS Service Quotas Generator](../../codegen/quotas/)
 
 
 ### Example code
@@ -25,26 +28,21 @@ Use the module in your Terraform code, replacing `<VERSION>` with the latest ver
 page](https://github.com/gruntwork-io/terraform-aws-utilities/releases):
 
 ```hcl
-module "path" {
+module "quota_increase" {
   source = "git::git@github.com:gruntwork-io/terraform-aws-utilities.git//modules/quota-increase?ref=<VERSION>"
 
-    request_quota_increase = {
-      nat_gateway = 40,
-      nacl_rules = 25
-    }
+  vpc_rules_per_network_acl              = 30
+  vpc_nat_gateways_per_availability_zone = 30
 }
 ```
 
-The argument to pass is:
-
-* `request_quota_increase`: A map with the desired resource and the new quota. The current supported resources are `nat_gateway` and `nacl_rules`. Feel free to contribute to this module to add support for more `quota_code` and `service_code` options in [main.tf](main.tf)!
-
+The [input variables](../../modules/request-quota-increase/variables.tf) for the module have been automatically generated using the [AWS Service Quotas Generator](../../codegen/quotas/). All adjustable Service Quotas are as separate input variables.
 
 When you run `apply`, the `new_quotas` output variable will confirm to you that a quota request has been made!
 
 ```hcl
 new_quotas = {
-  "nat_gateway" = {
+  "vpc_nat_gateways_per_availability_zone" = {
     "adjustable" = true
     "arn" = "arn:aws:servicequotas:us-east-1:<account-id>:vpc/L-FE5A380F"
     "default_value" = 5
@@ -72,7 +70,10 @@ aws service-quotas list-requested-service-quota-change-history --region <REGION>
 
 ### Finding out the Service Code and Quota Code
 
-When you need to add a new resource, you can check the available services with
+You can check adjustable quotas in the [input variables](../../modules/request-quota-increase/variables.tf).
+
+
+Alternatively, you can check the available services with
 
 ```
 aws service-quotas list-services --region <REGION> --output table
@@ -93,7 +94,7 @@ quota is 30 and you ask for a new quota of 25, this is the output:
 
 ```hcl
 new_quotas = {
-  "nat_gateway" = {
+  "vpc_nat_gateways_per_availability_zone" = {
     "adjustable" = true
       "arn" = "arn:aws:servicequotas:us-east-1:<account-id>:vpc/L-FE5A380F"
       "default_value" = 5