Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tests: cleanup and ci hardening #2397

Merged
merged 11 commits into from
Aug 16, 2020
Merged

Conversation

henryiii
Copy link
Collaborator

@henryiii henryiii commented Aug 15, 2020

Refactor the tests. Use a consistent failing/skipping scheme.

  • Attempt to allow 3.9-dev to fail (not sure how it works)
  • Fail-fast diagnostic removed, fails fast (once) with the correct error message without a custom wrapper.
  • Using xfail where possible (skip means not-applicable, xfail means it should ideally pass)
  • Use importorskip instead of ugly custom skips
  • PyTest 3.1+ required instead of 3.0 (current is 6.0 for Py3, 4.6.9 for Py2) - better warning reporting
  • Use info.py to simplify skips/xfails/checks
  • Update links to PyPy issues
  • Several (7 or so) skipped PyPy tests are now passing (especially those requiring NumPy) (reported by xfail!)
  • Strings used for the skipif/xfails in most cases so that PyTest reports the condition causing the skip/xfail directly in the report.

Skips/xfails are more consistent, no more pytest.bug_in_pypy, pytest.unsupported_on_pypy_lt_6, etc. (PyPy 7+ required, so ignoring check for old PyPy verisions). No more injection into the pytest module. xfails are reported at the end and most are strictly required. We can use reasons or raises to improve the test reporting.

@henryiii henryiii marked this pull request as draft August 16, 2020 02:47
@henryiii henryiii force-pushed the tests/cleanup branch 5 times, most recently from 3d37b12 to ec86a98 Compare August 16, 2020 04:24
tests/info.py Outdated Show resolved Hide resolved
tests/test_eval.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@bstaletic bstaletic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like this PR.

tests/pytest.ini Outdated Show resolved Hide resolved
tests/info.py Outdated Show resolved Hide resolved
tests/test_eval.py Show resolved Hide resolved
@henryiii henryiii marked this pull request as ready for review August 16, 2020 16:30
@henryiii
Copy link
Collaborator Author

henryiii commented Aug 16, 2020

I believe 7 or so tests are now passing in PyPy that used to fail (and were hidden by skip instead of xfail), mostly due to better NumPy support in recent PyPy's.

@YannickJadoul
Copy link
Collaborator

YannickJadoul commented Aug 16, 2020

I believe 7 or so tests are now passing in PyPy that used to fail (and were hidden by skip instead of xfail), mostly due to better NumPy support in recent PyPy's.

That's great to hear! :-)

@henryiii
Copy link
Collaborator Author

If I get two approvals, I'll squash and merge. :)

@henryiii henryiii changed the title Tests/cleanup tests: cleanup and ci hardening Aug 16, 2020
Copy link
Collaborator

@bstaletic bstaletic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great. Thanks for the PR.

Copy link
Collaborator

@rwgk rwgk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, thanks Henry!

@henryiii henryiii merged commit 4d9024e into pybind:master Aug 16, 2020
@henryiii henryiii deleted the tests/cleanup branch August 16, 2020 20:02
YannickJadoul added a commit to YannickJadoul/pybind11 that referenced this pull request Aug 16, 2020
rwgk pushed a commit that referenced this pull request Aug 17, 2020
@henryiii henryiii added the ci related to the CI system label Aug 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci related to the CI system
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants