test: add new integration test utility

[nathanwn]

This pull request adds a new integration test utility script that:

• Introduces a new YAML schema for specifying integration tests, which offers a few advantages over testing through bash. The script generally helps avoid certain kinds of mistakes observed previously while working with bash scripts, such as forgetting to enforce the exit code of a shell command.
• Streamlines the process of developing and running integration tests locally, which was suboptimal previously.
• Provides some level of fail-fast static validation for test cases to help save time during local development as well as CI in case unintentional mistakes happen when test cases are created or changed.
• Helps organize individual test cases into separate directories. This means the output of each test case, including the database, is separated from one another, which enables testing on specific states of the database.
• Allows testing not just the macaron Python package, but also any wrapper script of Macaron (including but not limited to the run_macaron.sh script).
• Allows selecting/deselecting a subset of test cases to run through the tagging mechanism. This is helpful for some purposes, such as sharing test cases between the Python package and the run_macaron.sh wrapper script, or disabling certain test cases when some services are unreachable.

This pull request also converts a large number of test cases from the old integration test setup to this new setup with the utility. Some conversions may conflict with some changes made in #748, but hopefully not be too hard to resolve; otherwise, we can selectively drop certain commits from this PR before merging and proceed with the conversion later.

Changes

The utility script and test data

The utility script and test data are under the tests/integration directory. Detailed instructions for the utility script can be found in the README.md file under the same directory.

Most test cases specified in YAML config are converted from the old integration_test.sh and integration_test_docker.sh script. For each converted test case, I made it a separate commit for easy review. Please refer to the list of commits when you review the conversion.

Changes to Makefile

Instead of completely replacing scripts/dev_script/{integration_tests,integration_test_docker}.sh, the new utility script is now called within both scripts, since this PR only converts parts of these scripts and not fully.

Since the {integration_tests,integration_test_docker}.sh are not fully replaced, no Makefile change is required.

Changes to GitHub Actions Workflows

The integration test utility script requires two dependencies already available in the development virtual environment: ruamel.yaml for YAML parsing and cfgv for human-readable YAML validation errors.

The script also requires Python 3.11, although compatibility with other Python versions has not been tested. Another pending pull request, #748, is also adding Python to the .github/workflows/_build_docker.yml workflow file, and so our change is similar here.

Changes to Documentation

Besides the ./tests/integration/README.md file, reference was made to the new integration test utility in the Writing a New Check section on the Macaron Developer's Guide website page.

Commits

The first commit in this PR includes the integration test utility, GitHub Actions workflow changes, and documentation changes. Each later PR converts a test case from the previous setup to this new setup.

[nathanwn]

Rebased to resolve conflicts with staging after all comments have been addressed.

[tromai]

LGTM. Thanks for the change.

[behnazh-w]

Rebased to resolve conflicts with staging after all comments have been addressed.

Thanks a lot 🙂