Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions sdks/python/apache_beam/yaml/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,28 @@ https://beam.apache.org/documentation/sdks/yaml/

For information about contributing to Beam YAML see
https://docs.google.com/document/d/19zswPXxxBxlAUmswYPUtSc-IVAu1qWvpjo1ZSDMRbu0

## Integration Tests

The integration_tests.py dynamically creates test methods based on the yaml
files provided in the `tests` and `extended_tests` directories and runs the
pipeline. It also contains context managers for setting up test environments for
both precommit tests (e.g. `tests` folder) and postcommit tests
(e.g. `extended_tests` folder).

To run the precommit tests:

```bash
pytest -v integration_tests.py

or

pytest -v integration_tests.py::<yaml_file_name_without_extension>Test
```

To run the postcommit tests:

```bash
pytest -v integration_tests.py --test_files_dir="extended_tests"
```

20 changes: 18 additions & 2 deletions sdks/python/apache_beam/yaml/extended_tests/postgres.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ pipelines:
url: "{TEMP_DB}"
query: "INSERT INTO tmp_table (value, rank) VALUES(?,?)"

# Postgres read pipeline
# Postgres read pipeline with driver_class_name
- pipeline:
type: chain
transforms:
Expand All @@ -50,4 +50,20 @@ pipelines:
- {value: 123, rank: 0}
- {value: 456, rank: 1}
- {value: 789, rank: 2}


# Postgres read pipeline without driver_class_name
# This is to prevent https://github.com/apache/beam/issues/35122 from
# happening again.
- pipeline:
type: chain
transforms:
- type: ReadFromPostgres
config:
url: "{TEMP_DB}"
query: "SELECT * FROM tmp_table"
- type: AssertEqual
config:
elements:
- {value: 123, rank: 0}
- {value: 456, rank: 1}
- {value: 789, rank: 2}
Loading