Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use unittest discover for running tests #515

Merged
merged 5 commits into from
May 12, 2020
Merged

Conversation

ievacerny
Copy link
Contributor

Closes #514

The PR adds a custom test loader functions that is used by unittest discover which allows excluding tests that match with a regex pattern specified through environment variable EXCLUDE_TESTS.

This is an attempt to simulate the behaviour of --exclude flag in nosetests.

Caveat: unittest attempts to import all test modules during loading, therefore, it will try to import test modules in ui/qt4 when running tests for wx. The import will fail and the loader will return a unittest.loader._FailedTest which has the path to the test module in its ID. Because the path contains excluded pattern qt the test will be filtered out.

@ievacerny ievacerny self-assigned this Apr 30, 2020
@ievacerny
Copy link
Contributor Author

I've been running the test suite in many different ways to check if the behaviour of unittest and nose is the same, but I'm struggling to get consistent results.

It might be due to test interactions. Investigating it now but it might take a while as it's not an easy thing to debug.

Copy link
Contributor

@corranwebster corranwebster left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, and seems equivalent to what nose does, so good from that point of view. We need to fix the imports done in pyface/__init__.py as they will be imported whenever pyface is imported for the first time as a side-effect.

I have some concerns about the "collect and then filter" approach, particularly if we are running the tests in an environment with both wxPython and a Qt installed. If we get unlucky in that situation then we may end up with a Qt app (or vice-versa a Wx app) up and running before the backend we really want gets initialized. But we can prevent that by importing pyface.toolkit as an early line in load_tests.

It might even be possible to use the pyface.toolkit value to select the filter rather than having to rely on an environment variable (but maybe still have the choice overridable by an envirionment variable for safety, if you do this). But isn't needed for this PR.

pyface/__init__.py Outdated Show resolved Hide resolved
pyface/util/testing.py Outdated Show resolved Hide resolved
pyface/__init__.py Show resolved Hide resolved
@corranwebster
Copy link
Contributor

Wx test failures are because:

  1. the linux wheel referenced in etstool.py is no longer there, and we need another one
  2. pip is installing wxPython 4.1 on mac and it has introduced test failures

@corranwebster
Copy link
Contributor

With #510 merged, this needs a merge of master which will hopefully then make the tests pass.

@codecov-io
Copy link

Codecov Report

Merging #515 into master will increase coverage by 0.11%.
The diff coverage is 94.73%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #515      +/-   ##
==========================================
+ Coverage   36.84%   36.95%   +0.11%     
==========================================
  Files         470      470              
  Lines       25921    25951      +30     
  Branches     3931     3937       +6     
==========================================
+ Hits         9550     9590      +40     
+ Misses      15955    15943      -12     
- Partials      416      418       +2     
Impacted Files Coverage Δ
pyface/ui/qt4/workbench/split_tab_widget.py 12.47% <50.00%> (ø)
pyface/__init__.py 82.60% <87.50%> (+11.18%) ⬆️
pyface/qt/QtCore.py 100.00% <100.00%> (ø)
pyface/qt/QtGui.py 100.00% <100.00%> (ø)
pyface/qt/QtMultimedia.py 100.00% <100.00%> (ø)
pyface/qt/QtMultimediaWidgets.py 100.00% <100.00%> (ø)
pyface/qt/QtNetwork.py 100.00% <100.00%> (ø)
pyface/qt/QtOpenGL.py 100.00% <100.00%> (ø)
pyface/qt/QtSvg.py 100.00% <100.00%> (ø)
pyface/qt/QtTest.py 100.00% <100.00%> (ø)
... and 13 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b149f5f...c3c825b. Read the comment docs.

@ievacerny
Copy link
Contributor Author

The tests on Travis seem to work fine so I'm merging this PR...

@ievacerny ievacerny merged commit abcdb6a into master May 12, 2020
@ievacerny ievacerny deleted the tst/514-stop-using-nose branch May 12, 2020 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Stop using Nose
3 participants