Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: move collection of flaky tests out of pytest_no_flaky #133

Merged
merged 1 commit into from
Aug 4, 2023
Merged

chore: move collection of flaky tests out of pytest_no_flaky #133

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 11 additions & 5 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -250,16 +250,22 @@ pytest_macOS_for_GitHub:
--randomly-dont-reset-seed \
${PYTEST_OPTIONS}

# Warning: collecting tests can trigger the parameter_search flaky test
# FIXME: https://github.com/zama-ai/concrete-ml-internal/issues/3812
.PHONY: check_currentl_flaky_tests # Print the current list of known flaky tests
check_current_flaky_tests:
echo "Skip the following known flaky tests (test file: number of skipped configs):"
poetry run pytest ./tests --collect-only -m flaky -qq

# --ignore makes pytest ignore complete files (because all tests are flaky). This needs to be done
# even though the associated module are marked as "flaky" because this does not seem to work in
# some cases
# even though the associated module is marked as "flaky" because collecting the test is enough to
# trigger the paremeter_search error
# Remove the --ignore and include 'check_current_flaky_tests' once the flaky is fixed
# FIXME: https://github.com/zama-ai/concrete-ml-internal/issues/3812
.PHONY: pytest_no_flaky # Run pytest but ignore known flaky issues (so no coverage as well)
pytest_no_flaky:
poetry run pytest --version
echo "Skip the following known flaky tests (test file: number of skipped configs):"
poetry run pytest ./tests --collect-only -m flaky -qq
echo "Warning: coverage is disabled"
echo "Warning: known flaky tests are skipped and coverage is disabled"
poetry run pytest tests/ \
--durations=10 -svv \
--capture=tee-sys \
Expand Down