Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion contributor-docs/python-tips.md
Original file line number Diff line number Diff line change
Expand Up @@ -265,7 +265,17 @@ Execute the following code for running tests using tox:

### Running Tests Using gradle

Integration tests suites on Jenkins are configured in groovy files that launch certain gradle tasks ([example](https://github.com/apache/beam/blob/0fd6a044df5b9f26d567e0f9a619a665a0f4043b/.test-infra/jenkins/job_PostCommit_Python.groovy#L43)). You could launch test suites locally by executing the gradle targets directly (for example: `./gradlew :sdks:python:test-suites:dataflow:py39:postCommitPy39`). This option may only be available to committers, as by default the test suites are configured to use the [`apache-beam-testing`](https://github.com/apache/beam/blob/0fd6a044df5b9f26d567e0f9a619a665a0f4043b/sdks/python/scripts/run_integration_test.sh#L70) project.
Integration tests suites on Jenkins are configured in groovy files that launch certain gradle tasks ([example](https://github.com/apache/beam/blob/0fd6a044df5b9f26d567e0f9a619a665a0f4043b/.test-infra/jenkins/job_PostCommit_Python.groovy#L43)). You could launch test suites locally by executing the gradle targets directly (for example: `./gradlew :sdks:python:test-suites:dataflow:py39:postCommitPy39`). This option may only be available to committers, as by default the test suites are configured to use the [`apache-beam-testing`](https://github.com/apache/beam/blob/0fd6a044df5b9f26d567e0f9a619a665a0f4043b/sdks/python/scripts/run_integration_test.sh#L70) project.

### Environment Variables for Test Stability

The following environment variables can be used to improve test stability in CI environments:

**Test execution settings:**
- `PYTEST_XDIST_WORKER_COUNT=1` - Force sequential test execution
- `PYTHONHASHSEED=0` - Ensure deterministic hash behavior
- `OMP_NUM_THREADS=1` - Limit OpenMP threads
- `OPENBLAS_NUM_THREADS=1` - Limit OpenBLAS threads

To run only a subset of tests using this approach, you could adjust the test label in the test (such as [it_postcommit](https://github.com/apache/beam/blob/25e6008e8919c2f31eaebae2662b44e02f9f37a1/sdks/python/apache_beam/io/gcp/pubsub_integration_test.py#L211)) and the [selector](https://github.com/apache/beam/blob/25e6008e8919c2f31eaebae2662b44e02f9f37a1/sdks/python/test-suites/dataflow/common.gradle#L117) where the test suite is defined.

Expand Down
2 changes: 1 addition & 1 deletion sdks/python/scripts/run_pytest.sh
Original file line number Diff line number Diff line change
Expand Up @@ -152,4 +152,4 @@ if [[ $status1 != 0 && $status1 != 5 ]]; then
fi
if [[ $status2 != 0 && $status2 != 5 ]]; then
exit $status2
fi
fi
5 changes: 2 additions & 3 deletions sdks/python/test-suites/tox/common.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -34,10 +34,9 @@ test.dependsOn "testPy${pythonVersionSuffix}Dill"

// toxTask "testPy${pythonVersionSuffix}Dask", "py${pythonVersionSuffix}-dask", "${posargs}"
// test.dependsOn "testPy${pythonVersionSuffix}Dask"
// Since codecoverage reports will always be generated for py39,
// all tests will be exercised.
project.tasks.register("preCommitPy${pythonVersionSuffix}") {
// Since codecoverage reports will always be generated for py39,
// all tests will be exercised.
// dependsOn = ["testPy${pythonVersionSuffix}Cloud", "testPython${pythonVersionSuffix}"]
dependsOn = ["testPy${pythonVersionSuffix}Cloud", "testPython${pythonVersionSuffix}"]

}
Loading