Refactor integration tests into matrix #3618
Conversation
| run: | | ||
| RUN_PRIVATE=$IS_NOT_FORK LUDWIG_TEST_SUITE_TIMEOUT_S=7200 pytest -v --timeout 300 --durations 100 -m "not slow and not combinatorial and not horovod and not llm and not integration_tests_a and not integration_tests_b and not integration_tests_c and not integration_tests_d" --junitxml pytest.xml tests/integration_tests |
There was a problem hiding this comment.
@justinxzhao why is it that only test suite E had all these not integration_tests_a ... markers but the others don't?
There was a problem hiding this comment.
Ah, this is because tests for groups a/b/c/d, are explicitly marked with pytest.mark while the last bucket (in this case "e"), is a catch-all for all of the rest of unmarked tests. The thinking here was that folks adding a new file in integration_tests/ shouldn't have to think about assigning a marker.
| run: | | ||
| RUN_PRIVATE=$IS_NOT_FORK LUDWIG_TEST_SUITE_TIMEOUT_S=7200 pytest -v --timeout 300 --durations 100 -m "not slow and not combinatorial and not horovod and not llm and not integration_tests_a and not integration_tests_b and not integration_tests_c and not integration_tests_d" --junitxml pytest.xml tests/integration_tests |
There was a problem hiding this comment.
Ah, this is because tests for groups a/b/c/d, are explicitly marked with pytest.mark while the last bucket (in this case "e"), is a catch-all for all of the rest of unmarked tests. The thinking here was that folks adding a new file in integration_tests/ shouldn't have to think about assigning a marker.
| - "integration_tests_b" | ||
| - "integration_tests_c" | ||
| - "integration_tests_d" | ||
| - "integration_tests_e" |
There was a problem hiding this comment.
This should work if we want to keep the markers unchanged:
# catch-all
- "not integration_tests_a and not integration_tests_b and not integration_tests_c and not integration_tests_d"
No description provided.