support for running all the test cases

[macnev2013]

Motivation

We are using terraform test suite as a quality gate for each PR, but flakiness of the suite is causing a lot of frictions for the dev team.

For us, the parity metrics of the terraform test suite provides more value than using it as a quality gate. So we have decided to externalise the test suite and schedule it to run at a certain time interval.

Features

• Python CLI tool:
  • Patches the terraform repository
  • Builds testing binaries for localstack community/pro/all supported services based on parameters
• Pytest plugin to list and execute test cases written in golang
• Pytest hook that launches LocalStack with a certain Docker image depending on the argument
• Github actions
  • Executes all test for all services in concurrent runner
  • Publishes test results in workflow's summary
  • Scheduled test execution (once a week)

[thrau]

overall those are very cool changes!

my two biggest points are:

1. it doesn't work on my local machine. probably missing some docs on how to build the binary.
2. code and project organization could be improved by making it consistent with our other python projects (concrete comments inline)

Functionality

i tested the readme and the collection works

 % python -m pytest --co | head
============================= test session starts ==============================
platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0
rootdir: /home/thomas/workspace/localstack/localstack-terraform-test
collected 7448 items

<GoFile terraform-provider-aws/internal/acctest/provider_test.go>
  <GoItem TestAccProvider_DefaultTags_emptyBlock>
  <GoItem TestAccProvider_DefaultTagsTags_none>
  <GoItem TestAccProvider_DefaultTagsTags_one>
  <GoItem TestAccProvider_DefaultTagsTags_multiple>

when running tests, however, i get some errors:

thomas@om ~/workspace/localstack/localstack-terraform-test
 % python -m pytest terraform-provider-aws/internal/service/sqs           
============================================================================================================= test session starts =============================================================================================================
platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0
rootdir: /home/thomas/workspace/localstack/localstack-terraform-test
collected 35 items                                                                                                                                                                                                                            

terraform-provider-aws/internal/service/sqs/queue_data_source_test.go FF                                                                                                                                                                [  5%]
terraform-provider-aws/internal/service/sqs/queue_policy_test.go FFFF                                                                                                                                                                   [ 17%]
terraform-provider-aws/internal/service/sqs/queue_redrive_allow_policy_test.go FFFF                                                                                                                                                     [ 28%]
terraform-provider-aws/internal/service/sqs/queue_redrive_policy_test.go FFFF                                                                                                                                                           [ 40%]
terraform-provider-aws/internal/service/sqs/queue_test.go FFFFFFFFFFFFFFFFFFFFF                                                                                                                                                         [100%]

================================================================================================================== FAILURES ===================================================================================================================
_________________________________________________________________________________________________ Test Case: TestAccSQSQueueDataSource_basic __________________________________________________________________________________________________
Failure Reason:
sh: 1: ./test-bin/sqs.test: not found

__________________________________________________________________________________________________ Test Case: TestAccSQSQueueDataSource_tags __________________________________________________________________________________________________
Failure Reason:
sh: 1: ./test-bin/sqs.test: not found

___________________________________________________________________________________________________ Test Case: TestAccSQSQueuePolicy_basic ____________________________________________________________________________________________________
Failure Reason:
sh: 1: ./test-bin/sqs.test: not found

i assume this is because i need the test binary, but i don't know how to build it using the instructions given to me.

[macnev2013]

Hey @thrau ,

1. it doesn't work on my local machine. probably missing some docs on how to build the binary.

Sorry missed to add this in the readme earlier.

• You need to run python -m terraform_pytest.main build -s all to build the testing binary or you can build the binary for specific service by python -m terraform_pytest.main build -s s3 before running the test cases with pytest.

Let me know how it goes.

[thrau]

hey! just testing the open PR. i can't seem to build the go test binary. any idea what's going on? i feel this part of the pipeline should be fairly reliable.

after adding some more log output (unfortunately the line 156 in utils.py swallows all the output), i got this:

thomas@om ~/workspace/localstack/localstack-terraform-test
 % python -m terraform_pytest.main build -s s3
Building s3...
Traceback (most recent call last):
  File "/home/thomas/.pyenv/versions/3.10.8/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/thomas/.pyenv/versions/3.10.8/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/thomas/workspace/localstack/localstack-terraform-test/terraform_pytest/main.py", line 56, in <module>
    cli()
  File "/home/thomas/workspace/localstack/localstack-terraform-test/.venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/thomas/workspace/localstack/localstack-terraform-test/.venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/thomas/workspace/localstack/localstack-terraform-test/.venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/thomas/workspace/localstack/localstack-terraform-test/.venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/thomas/workspace/localstack/localstack-terraform-test/.venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/thomas/workspace/localstack/localstack-terraform-test/terraform_pytest/main.py", line 42, in build
    build_test_bin(service=service, tf_root_path=realpath(TF_REPO_NAME))
  File "/home/thomas/workspace/localstack/localstack-terraform-test/terraform_pytest/utils.py", line 156, in build_test_bin
    raise Exception(f"Error while building test binary for {service}: {stdout}")
Exception: Error while building test binary for s3: go: github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/v2@v2.0.0-beta.18 requires
	github.com/aws/aws-sdk-go@v1.42.52: missing go.sum entry; to add it:
	go mod download github.com/aws/aws-sdk-go

after going into the terraform provider submodule and running go mod download github.com/aws/aws-sdk-go
it seems to work.

but maybe it would be good to be sure that the command runs without errors on a user's machine?

you can also tell me that you think this is unimportant since this will always run in CI. happy to argue 😄

[alexrashed]

This is really an awesome set of changes. I was a bit nitpicky with the review, but a lot of the comments are just comments for the next iteration of the project.
I'm really looking forward to having the tests and their parity data being run and collected automatically. 🚀

[macnev2013]

hey! just testing the open PR. i can't seem to build the go test binary. any idea what's going on? i feel this part of the pipeline should be fairly reliable.

after adding some more log output (unfortunately the line 156 in utils.py swallows all the output), i got this:

thomas@om ~/workspace/localstack/localstack-terraform-test
 % python -m terraform_pytest.main build -s s3
Building s3...
Traceback (most recent call last):
  File "/home/thomas/.pyenv/versions/3.10.8/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/thomas/.pyenv/versions/3.10.8/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/thomas/workspace/localstack/localstack-terraform-test/terraform_pytest/main.py", line 56, in <module>
    cli()
  File "/home/thomas/workspace/localstack/localstack-terraform-test/.venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/thomas/workspace/localstack/localstack-terraform-test/.venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/thomas/workspace/localstack/localstack-terraform-test/.venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/thomas/workspace/localstack/localstack-terraform-test/.venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/thomas/workspace/localstack/localstack-terraform-test/.venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/thomas/workspace/localstack/localstack-terraform-test/terraform_pytest/main.py", line 42, in build
    build_test_bin(service=service, tf_root_path=realpath(TF_REPO_NAME))
  File "/home/thomas/workspace/localstack/localstack-terraform-test/terraform_pytest/utils.py", line 156, in build_test_bin
    raise Exception(f"Error while building test binary for {service}: {stdout}")
Exception: Error while building test binary for s3: go: github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/v2@v2.0.0-beta.18 requires
	github.com/aws/aws-sdk-go@v1.42.52: missing go.sum entry; to add it:
	go mod download github.com/aws/aws-sdk-go

after going into the terraform provider submodule and running go mod download github.com/aws/aws-sdk-go it seems to work.

but maybe it would be good to be sure that the command runs without errors on a user's machine?

you can also tell me that you think this is unimportant since this will always run in CI. happy to argue 😄

@thrau I've tried to reproduce the issue in my system as well as some other systems. Seems like its working fine for me. Still I'm adding a blind fix which should take care of it.

[alexrashed]

Thanks for addressing all the PR comments!
I only added two small comments (small nitpicks): A small enhancement to the README and the deletion of the compose file.
From my perspective, we're good to go with the first iteration! 🚀
I tested it locally on my machine with STS (which worked fine). @thrau maybe we could address your issues with the next iteration?