Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci(datasets): Run tensorflow tests separately from other dataset tests #377

Merged
merged 11 commits into from
Oct 6, 2023
Merged

ci(datasets): Run tensorflow tests separately from other dataset tests #377

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
1c6a88d
Add partitioned and incremental datasets to repo
merelcht Oct 5, 2023
2ad6b14
Run only tensorflow tests
merelcht Oct 5, 2023
99a2ee8
Run only tensorflow tests
merelcht Oct 5, 2023
aefd444
Run tensorflow datasets separately
merelcht Oct 5, 2023
fbd72d2
Ignore tensorflow tests in coverage
merelcht Oct 5, 2023
c6e4a8f
Don't check coverage for only tensorflow tests
merelcht Oct 5, 2023
14f1365
Remove new datasets again, to keep this PR to build changes only
merelcht Oct 5, 2023
5030e35
Clean up
merelcht Oct 5, 2023
a46975a
Merge branch 'main' into migrate-partitioned-incremental-datasets
merelcht Oct 6, 2023
43b04f6
Add comment to explain why tensorflow tests are run separately
merelcht Oct 6, 2023
cd84746
Merge branch 'main' into migrate-partitioned-incremental-datasets
merelcht Oct 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions .github/workflows/unit-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,9 +49,12 @@ jobs:
pip install ".[test]"
- name: pip freeze
run: pip freeze
- name: Run unit tests for Linux / all plugins
if: inputs.os != 'windows-latest'
- name: Run unit tests for Linux / kedro-airflow, kedro-docker, kedro-telemetry
if: inputs.os != 'windows-latest' && inputs.plugin != 'kedro-datasets'
run: make plugin=${{ inputs.plugin }} test
- name: Run unit tests for Linux / kedro-datasets
if: inputs.os != 'windows-latest' && inputs.plugin == 'kedro-datasets'
run: make dataset-tests
Comment on lines +52 to +57
Copy link
Contributor

@SajidAlamQB SajidAlamQB Oct 6, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should leave a comment why this was split.

- name: Run unit tests for Windows / kedro-airflow, kedro-docker, kedro-telemetry
if: inputs.os == 'windows-latest' && inputs.plugin != 'kedro-datasets'
run: |
Expand Down
5 changes: 5 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,11 @@ lint:
test:
cd $(plugin) && pytest tests --cov-config pyproject.toml --numprocesses 4 --dist loadfile

# Run test_tensorflow_model_dataset separately, because these tests are flaky when run as part of the full test-suite
dataset-tests:
cd kedro-datasets && pytest tests --cov-config pyproject.toml --numprocesses 4 --dist loadfile --ignore tests/tensorflow
cd kedro-datasets && pytest tests/tensorflow/test_tensorflow_model_dataset.py --no-cov

test-sequential:
cd $(plugin) && pytest tests --cov-config pyproject.toml

Expand Down
2 changes: 1 addition & 1 deletion kedro-datasets/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ version = {attr = "kedro_datasets.__version__"}
[tool.coverage.report]
fail_under = 100
show_missing = true
omit = ["tests/*", "kedro_datasets/holoviews/*", "kedro_datasets/snowflake/*"]
omit = ["tests/*", "kedro_datasets/holoviews/*", "kedro_datasets/snowflake/*", "kedro_datasets/tensorflow/*"]
exclude_lines = ["pragma: no cover", "raise NotImplementedError"]

[tool.pytest.ini_options]
Expand Down