chore: wrap up integration tests conversion (#782)

Signed-off-by: Trong Nhan Mai <trong.nhan.mai@oracle.com>

Author: Trong Nhan Mai

Makefile

@@ -298,16 +298,38 @@ test-go:
 # Note: to disable npm tests set `NO_NPM` environment variable to `TRUE`.
 .PHONY: integration-test
 integration-test:
-	scripts/dev_scripts/integration_tests.sh $(REPO_PATH) "${HOME}"
+	if [ "${NO_NPM}" == "TRUE" ]; then \
+    	echo "Note: NO_NPM environment variable is set to TRUE, so npm tests will be skipped."; \
+		python ./tests/integration/run.py \
+			run \
+			--include-tag macaron-python-package \
+			--exclude-tag skip \
+			--exclude-tag npm-registry-testcase \
+			./tests/integration/cases/...; \
+	else \
+		python ./tests/integration/run.py \
+			run \
+			--include-tag macaron-python-package \
+			--exclude-tag skip \
+			./tests/integration/cases/...; \
+	fi
 
 .PHONY: integration-test-docker
 integration-test-docker:
-	scripts/dev_scripts/integration_tests_docker.sh $(REPO_PATH) scripts/release_scripts/run_macaron.sh
+	python ./tests/integration/run.py \
+		run \
+		--macaron scripts/release_scripts/run_macaron.sh \
+		--include-tag macaron-docker-image \
+		--exclude-tag skip \
+		./tests/integration/cases/...
 
 # Update the expected results of the integration tests after generating the actual results.
 .PHONY: integration-test-update
 integration-test-update:
-	scripts/dev_scripts/integration_tests.sh $(REPO_PATH) "${HOME}" "--update"
+	python ./tests/integration/run.py \
+		update \
+		--exclude-tag skip \
+		./tests/integration/cases/...
 
 # Build a source distribution package and a binary wheel distribution artifact.
 # When building these artifacts, we need the environment variable SOURCE_DATE_EPOCH

scripts/dev_scripts/integration_tests.sh

@@ -1,2 +1,2 @@
-# Copyright (c) 2023 - 2023, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2022 - 2024, Oracle and/or its affiliates. All rights reserved.
 # Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/.

scripts/dev_scripts/integration_tests_docker.sh

@@ -133,38 +133,53 @@ $ python ./tests/integration/run.py run ./all/cases/...
 
 In certain cases, we can utilize the feature of tags to select a subset of test cases to run with the `run` command.
 
-Each test case can be attached with one or more tags in the yaml configuration. For example, you may find some of our test cases having the `docker` tag as follows.
+Each test case can be attached with one or more tags in the yaml configuration. For example, you may find some of our test cases having the tags as follows.
 
 ```yaml
 description: ...
 tags:
-- docker
+- macaron-python-package
+- macaron-docker-image
 steps:
 - ...
 ```
 
-We typically have the test cases for the container image being a subset of the test cases for the Macaron Python package. We can mark the test cases shared for both purposes with the `docker` tag. When we do integration testing for the container image, we can add the argument `--include-tag docker` to filter only test cases tagged with `docker`.
+We typically have test cases that are shared for the container image and the Macaron Python package. We can mark the test cases shared for both purposes with `macaron-python-package` and `macaron-docker-image` tags.
+When we do integration testing for the container image, we can add the argument `--include-tag macaron-docker-image` to filter test cases that are tagged with `macaron-docker-image`.
 
 ```bash
-# Test the container image with test cases having the `docker` tag.
-$ python ./tests/integration/run.py run --include-tag docker ./all/cases/...
+# Test the container image with test cases having the `macaron-docker-image` tag.
+$ python ./tests/integration/run.py run --include-tag macaron-docker-image ./all/cases/...
 ```
 
-The `--include-tag` flag can be specified multiple times. A selected test case must contain all tags specified with the `--include-tag` flag.
+We can do the same with `macaron-python-package` when we do integration tests for the Macaron Python package.
+
+The `--include-tag` flag can be specified multiple times. A selected test case must be tagged with at least a tag specified with any of the `--include-tag` flags.
 
 ```bash
-# Test the container image with test cases having the `docker` tag.
+# Test the container image with test cases having EITHER `tag-a` for `tag-b` tag.
 $ python ./tests/integration/run.py run --include-tag tag-a --include-tag tag-b ./all/cases/...
 ```
 
 There is also the `--exclude-tag` flag. A selected test case must also not contain any tag specified with the `--exclude-tag` flag.
 
 ```bash
-# Only run test cases not tagged with `npm`.
-$ python ./tests/integration/run.py run --exclude-tag npm ./all/cases/...
+# Only run test cases not tagged with `npm-registry-testcase`.
+$ python ./tests/integration/run.py run --exclude-tag npm-registry-testcase ./all/cases/...
 ```
 
-You can simply think of each `--include-tag`/`--exclude-tag` argument as adding an additional constraint that a selected test case must satisfy".
+You can simply think of each `--include-tag`/`--exclude-tag` argument as adding an additional constraint that a selected test case must satisfy.
+
+Instructions on how to tag a test case for our CI/CD pipeline:
+- If you want a test case to **only** run for the container image, use **only** `macaron-docker-image`.
+- If you want a test case to **only** run with the Macaron Python package, use **only** `macaron-python-package`.
+- To skip a test case, use `skip`. `skip` still has the same effect if it's used with other tags.
+- If you want to run a test case for both the Macaron Python package and the docker container, use `macaron-python-package` and `macaron-docker-image` tags.
+- If you want to run test cases that must contain all of a given set of tags (e.g. `['tag-a', 'tag-b']`), please create an additional tag for those test cases (e.g `tag-a-b`) and use it within `--include-tag`.
+- Test cases marked with `npm-registry-testcase` are not run if the environment variable `NO_NPM` is set to `TRUE`. This only applies when you run the integration tests with:
+```bash
+$ make integration-test
+```
 
 ### Debug utility script
 

tests/e2e/repo_finder/__init__.py renamed to tests/analyze_json_output/__init__.py

@@ -4,6 +4,9 @@
 description: |
   Running the analysis with all checks excluded. This test should return an error code.
 
+tags:
+- macaron-python-package
+
 steps:
 - name: Run macaron analyze
   kind: analyze

tests/e2e/compare_e2e_result.py renamed to tests/analyze_json_output/compare_analyze_json_output.py

@@ -0,0 +1,16 @@
+# Copyright (c) 2024 - 2024, Oracle and/or its affiliates. All rights reserved.
+# Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/.
+
+description: |
+  Analyzing dependencies when neither the repository nor SBOM is available. We expect the analysis to finish with no errors.
+
+tags:
+- macaron-python-package
+
+steps:
+- name: Run macaron analyze
+  kind: analyze
+  options:
+    command_args:
+    - -purl
+    - pkg:maven/private.apache.maven/maven@4.0.0-alpha-1-SNAPSHOT?type=pom