support for running all the test cases

.github/workflows/main.yml

@@ -0,0 +1,88 @@
+on:
+  schedule:
+    - cron: '0 19 * * 6'
+  workflow_dispatch:
+    inputs:
+      services:
+        default: 'ls-community'
+        type: string
+        description: name of the service to execute tests for (e.g. "ls-community", "ls-pro", "ls-all", "s3,iam,ec2")
+      localstack-image:
+        required: false
+        type: string
+        default: 'localstack/localstack:latest'
+        description: localstack docker image name to test against
+# disabled pipeline for push events, as we want to run the pipeline only on a schedule or manually
+#  push:
+#    branches:
+#      - pytest-plugin
+
+name: Terraform Tests
+jobs:
+
+  prepare_list:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+        with:
+          submodules: 'true'
+      - id: set-matrix
+        run: echo "matrix=$(python -m terraform_pytest.get-services ${{ github.event.inputs.services || 'ls-community' }})" >> $GITHUB_OUTPUT
+    outputs:
+      matrix: ${{ steps.set-matrix.outputs.matrix }}
+
+  test_service:
+    needs: prepare_list
+    strategy:
+      max-parallel: 10
+      fail-fast: false
+      matrix:
+        service: ${{ fromJson(needs.prepare_list.outputs.matrix) }}
+    runs-on: ubuntu-latest
+
+    steps:
+
+    - uses: actions/checkout@v3
+      with:
+        submodules: 'true'
+
+    - uses: actions/setup-go@v3
+      with:
+        go-version: '1.18.x'
+        cache: true
+        cache-dependency-path: terraform-provider-aws/go.sum
+
+    - name: Set up Python 3.10.5
+      uses: actions/setup-python@v4
+      with:
+        python-version: '3.10.5'
+        cache: 'pip'
+
+    - name: Install system dependencies
+      run: |
+        sudo apt update --fix-missing
+        sudo apt install libsasl2-dev -y
+        pip install --upgrade pip
+        pip install -r requirements.txt
+
+    - name: Patch Terraform Provider
+      run: |
+        cd terraform-provider-aws && go mod vendor
+        cd ../
+        python -m terraform_pytest.main patch
+
+    - name: Build ${{ matrix.service }} Binary
+      run: |
+        python -m terraform_pytest.main build -s ${{ matrix.service }}
+        ls -la terraform-provider-aws/test-bin
+
+    - name: Run ${{ matrix.service }} Tests
+      run: |
+        python -m pytest --junitxml=target/reports/pytest.xml terraform-provider-aws/internal/service/${{ matrix.service }} -s -v --ls-start --ls-image ${{ github.event.inputs.localstack-image || 'localstack/localstack:latest' }}
+
+    - name: Publish ${{ matrix.service }} Test Results
+      uses: EnricoMi/publish-unit-test-result-action@v2
+      if: always()
+      with:
+        junit_files: target/reports/*.xml
+        check_name: ${{ matrix.service }} Terraform Test Results

.gitignore

@@ -1,10 +1,8 @@
-*~
-*.swp
-*.log
-*.bak
-
-.vscode
+.DS_Store
 .idea
-*.iml
-
-build/
+.venv
+.pytest_cache
+__pycache__
+target
+**/*.test
+report.xml

.gitmodules

@@ -1,9 +1,3 @@
-[submodule "localstack"]
-	path = localstack
-	url = https://github.com/localstack/localstack.git
 [submodule "terraform-provider-aws"]
 	path = terraform-provider-aws
 	url = https://github.com/hashicorp/terraform-provider-aws.git
-[submodule "moto"]
-	path = moto
-	url = git@github.com:localstack/moto

Makefile

@@ -0,0 +1,26 @@
+#!/bin/bash
+
+VENV_BIN ?= python3 -m venv
+VENV_DIR ?= .venv
+PIP_CMD ?= pip3
+
+ifeq ($(OS), Windows_NT)
+	VENV_ACTIVATE = $(VENV_DIR)/Scripts/activate
+else
+	VENV_ACTIVATE = $(VENV_DIR)/bin/activate
+endif
+
+$(VENV_ACTIVATE):
+	test -d $(VENV_DIR) || $(VENV_BIN) $(VENV_DIR)
+	$(VENV_RUN); $(PIP_CMD) install --upgrade pip setuptools wheel plux
+	touch $(VENV_ACTIVATE)
+
+VENV_RUN = . $(VENV_ACTIVATE)
+
+venv: $(VENV_ACTIVATE)    ## Create a new (empty) virtual environment
+
+install:
+	$(VENV_RUN); $(PIP_CMD) install -r requirements.txt
+
+format:
+	$(VENV_RUN); python -m isort .; python -m black .

README.md

@@ -1,61 +1,35 @@
-Terraform vs LocalStack
-=======================
-
-This repository contains scripts and CI configurations to to run the Terraform Acceptance test suite of the AWS provider against LocalStack
-
-## Utilities
-
-Some utilities for local development:
-
-* `bin/list-tests [--all]`: list the available tests by parsing the go test files.
-* `bin/install-aws-test` creates the binary for running the test suite (and installs it into `$HOME/.cache/localstack/aws.test`. requires go 1.16
-* `bin/run-tests [test]` run a specific test. this installs and runs localstack in a background process. add the flag `-t` to test against an already running localstack instance.
-
-## Finding and running tests
-
-After running `bin/install-aws-test`, use `bin/run-tests [OPTIONS...] [TESTS...]` to run individual tests or entire test suites.
-
-Here are some examples:
-
-* `bin/run-tests TestAccAWSAPIGatewayResource`
-* `bin/run-tests -t TestAccAWSAPIGatewayResource`: same as above, but does not start localstack
-* `bin/run-tests TestAccAWSAPIGateway`: runs all tests that match `TestAccAWSAPIGateway` (run `bin/list-tests TestAccAWSAPIGateway` to see which ones will be executed)
-* `bin/run-tests -e TestAccAWSAPIGatewayV2 TestAccAWSAPIGateway`: same as above, but excludes all tests that match `TestAccAWSAPIGatewayV2`.
-* `bin/run-tests -i localstack-tests.incl.txt`: runs all tests listed in the text file
-
-You can use `bin/list-tests` with the same parameters to see which tests will be executed,
-or to find specific tests based on patterns.
-
-For example:
-
-```
- % bin/list-tests Queue
-TestAccAWSBatchJobQueue
-TestAccAWSGameliftGameSessionQueue
-TestAccAWSMediaConvertQueue
-TestAccAWSSQSQueue
-TestAccAWSSQSQueuePolicy
-TestAccDataSourceAwsBatchJobQueue
-TestAccDataSourceAwsSqsQueue
-```
-
-or
-
-```
- % bin/list-tests "Data.*Queue"
-TestAccDataSourceAwsBatchJobQueue
-TestAccDataSourceAwsSqsQueue
-```
-
-## Generating the test reports
-
-Test logs are aggregated into `build/tests/*.log`, the command `bin/create-report` will create junit-like xml reports.
-These can then be rendered into html using `bin/create-report-html`, which also creates a summary page in `build/report.html`.
-For rendering html, you need `junit2html`.
-
-## Travis config
-
-### Build cache
-
-The Travis-CI worker caches the built `aws.test` binary across builds.
-The first build may therefore take a while.
+# Localstack Terraform Test Runner
+
+This is a test runner for localstack and terraform. It will run a test cases from the hashicrop [terraform provider aws](https://github.com/hashicorp/terraform-provider-aws.git) against Localstack Instance.
+
+Purpose of this project is to externalize the test cases from the localstack repo and run them against localstack to gather parity metrics.
+
+## Installation
+1. Clone the repository with submodules`
+2. Run `make venv` to create a virtual environment
+3. Run `make install` to install the dependencies
+
+## How to run?
+1. Run `python -m terraform_pytest.main patch` to apply the patch to the terraform provider aws
+2. Run `python -m terraform_pytest.main build -s s3` to build testing binary for the golang module
+3Now you are ready to use `python -m pytest` commands to list and run test cases from golang
+
+## How to run test cases?
+- To list down all the test case from a specific service, run `python -m pytest terraform-provider-aws/internal/service/<service> --collect-only -q`
+- To run a specific test case, run `python -m pytest terraform-provider-aws/internal/service/<service>/<test-file> -k <test-case-name> --ls-start` or `python -m pytest terraform-provider-aws/internal/service/<service>/<test-file>::<test-case-name> --ls-start`
+- Additional environment variables can be added by appending it in the start of the command, i.e. `AWS_ALTERNATE_REGION='us-west-2' python -m pytest terraform-provider-aws/internal/service/<service>/<test-file>::<test-case-name> --ls-start`
+
+## Default environment variables
+- **TF_ACC**: `1`
+- **AWS_ACCESS_KEY_ID**: `test`
+- **AWS_SECRET_ACCESS_KEY**: `test`
+- **AWS_DEFAULT_REGION**: `us-west-1`
+- **AWS_ALTERNATE_ACCESS_KEY_ID**: `test`
+- **AWS_ALTERNATE_SECRET_ACCESS_KEY**: `test`
+- **AWS_ALTERNATE_SECRET_ACCESS_KEY**: `test`
+- **AWS_ALTERNATE_REGION**: `us-east-2`
+- **AWS_THIRD_REGION**: `eu-west-1`
+
+## Options
+- `--ls-start`: Start localstack instance before running the test cases
+- `--ls-image`: Specify the localstack image to use, default is `localstack/localstack:latest`