Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ repos:
hooks:
- id: integration-test-vet
name: validate integration test cases
entry: python3
entry: python
args:
- ./tests/integration/run.py
- vet
Expand Down
2 changes: 1 addition & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ make docs
This command will build and generate the documentation into `docs/_build/html`. To view it locally, run (with the dev environment activated):

```
python3 -m http.server -d docs/_build/html
python -m http.server -d docs/_build/html
```

## Extend the API reference
Expand Down
2 changes: 1 addition & 1 deletion scripts/dev_scripts/integration_tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -397,7 +397,7 @@ run_macaron_clean $ANALYZE -purl pkg:maven/io.github.behnazh-w.demo/example-mave

check_or_update_expected_output $COMPARE_DEPS $DEP_RESULT $DEP_EXPECTED || log_fail

python3 ./tests/integration/run.py run \
python ./tests/integration/run.py run \
./tests/integration/cases/... || log_fail

# Important: This should be at the end of the file
Expand Down
2 changes: 1 addition & 1 deletion scripts/dev_scripts/integration_tests_docker.sh
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ python $COMPARE_POLICIES $POLICY_RESULT $POLICY_EXPECTED || log_fail
# Clean up and remove the virtual environment.
rm -rf "$VIRTUAL_ENV_PATH"

python3 ./tests/integration/run.py run \
python ./tests/integration/run.py run \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we would need to change the same for integration_tests.sh too.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure how I missed that :-)
See commit ba87342

--macaron scripts/release_scripts/run_macaron.sh \
--include-tag docker \
./tests/integration/cases/... || log_fail
Expand Down
22 changes: 11 additions & 11 deletions tests/integration/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
We have an integration test utility script, [`./tests/integration/run.py`](./run.py), for running integration tests. The script should be called within the dev virtual environment and from the root directory of the repository.

```bash
$ python3 ./tests/integration/run.py -h
$ python ./tests/integration/run.py -h
usage: ./tests/integration/run.py [-h] {check,vet,run,update} ...

positional arguments:
Expand All @@ -26,7 +26,7 @@ The utility offers 4 different commands, as shown in the help message above. Som

```bash
# Show help message for the check command.
$ python3 ./tests/integration/run.py check -h
$ python ./tests/integration/run.py check -h
```


Expand Down Expand Up @@ -81,21 +81,21 @@ You create a new test case by creating a new directory, then a `test.yaml` withi

```bash
# Schema-validate the ./test/case/directory/test.yaml file.
$ python3 ./tests/integration/run.py check ./test/case/directory
$ python ./tests/integration/run.py check ./test/case/directory
```

At this point, some expected result files do not exist yet, since you normally want to run `macaron` once, inspect the result files, then turn them into expected result files if they look good enough. To do this, you can run in **interactive** mode. In this mode, the utility stops at each step and asks if you want to run or skip a step. For `compare` steps, the utility also asks if you want to "update" the expected result file instead of compare.

```bash
# Run a test case in interactive mode.
$ python3 ./tests/integration/run.py run -i ./test/case/directory
$ python ./tests/integration/run.py run -i ./test/case/directory
```

After you have finished running the test case, you can rerun the test case to make sure everything works as expected.

```bash
# Run a test case end-to-end.
$ python3 ./tests/integration/run.py run ./test/case/directory
$ python ./tests/integration/run.py run ./test/case/directory
```

### Inspect test cases
Expand All @@ -104,7 +104,7 @@ Besides the interactive mode, the `run` command also has another special mode ca

```bash
# Run a test case in dry-run mode.
$ python3 ./tests/integration/run.py run -d ./test/case/directory
$ python ./tests/integration/run.py run -d ./test/case/directory
```

### Validate test cases before pushing commits to remote or running in CI
Expand All @@ -119,14 +119,14 @@ All commands (`check`, `vet`, `run`, and `update`) can process multiple test cas

```bash
# Run two test cases one after another.
$ python3 ./tests/integration/run.py run ./test_case_a/directory ./test_case_b/directory
$ python ./tests/integration/run.py run ./test_case_a/directory ./test_case_b/directory
```

You can also use the `...` path wildcard to allow for discovering test case directories recursively under a root directory.

```bash
# Run all test cases discovered recursively under a directory.
$ python3 ./tests/integration/run.py run ./all/cases/...
$ python ./tests/integration/run.py run ./all/cases/...
```

### Select a subset of test cases to run
Expand All @@ -147,21 +147,21 @@ We typically have the test cases for the container image being a subset of the t

```bash
# Test the container image with test cases having the `docker` tag.
$ python3 ./tests/integration/run.py run --include-tag docker ./all/cases/...
$ python ./tests/integration/run.py run --include-tag docker ./all/cases/...
```

The `--include-tag` flag can be specified multiple times. A selected test case must contain all tags specified with the `--include-tag` flag.

```bash
# Test the container image with test cases having the `docker` tag.
$ python3 ./tests/integration/run.py run --include-tag tag-a --include-tag tag-b ./all/cases/...
$ python ./tests/integration/run.py run --include-tag tag-a --include-tag tag-b ./all/cases/...
```

There is also the `--exclude-tag` flag. A selected test case must also not contain any tag specified with the `--exclude-tag` flag.

```bash
# Only run test cases not tagged with `npm`.
$ python3 ./tests/integration/run.py run --exclude-tag npm ./all/cases/...
$ python ./tests/integration/run.py run --exclude-tag npm ./all/cases/...
```

You can simply think of each `--include-tag`/`--exclude-tag` argument as adding an additional constraint that a selected test case must satisfy".
Expand Down
4 changes: 2 additions & 2 deletions tests/integration/run.py
Original file line number Diff line number Diff line change
Expand Up @@ -252,7 +252,7 @@ def cmd(self, macaron_cmd: str) -> list[str]:
result_file = self.options["result"]
expected_file = self.options["expected"]
return [
"python3",
"python",
os.path.abspath(os.path.join(*COMPARE_SCRIPTS[kind])),
*[result_file, expected_file],
]
Expand All @@ -276,7 +276,7 @@ def update_result(self, cwd: str) -> int:
if kind == "vsa":
proc = subprocess.run(
args=[
"python3",
"python",
os.path.abspath(os.path.join(*COMPARE_SCRIPTS[kind])),
"--update",
*[result_file, expected_file],
Expand Down