Remaining temporary files are reported repeatedly during tests

[cgohlke]

python Tests\run.py:

<snip>

running test_file_libtiff ...
..................=== remaining temporary files
c:\users\gohlke\appdata\local\temp\pillow-tests\temp_file_libtiff_13.tiff
c:\users\gohlke\appdata\local\temp\pillow-tests\temp_file_libtiff_6.tif
--------------------------------------------------------------------

----------------------------------------------------------------------
Ran 18 tests in 1.416s

OK
running test_file_libtiff_small ...
..................=== remaining temporary files
c:\users\gohlke\appdata\local\temp\pillow-tests\temp_file_libtiff_13.tiff
c:\users\gohlke\appdata\local\temp\pillow-tests\temp_file_libtiff_6.tif
c:\users\gohlke\appdata\local\temp\pillow-tests\temp_file_libtiff_small_13.tiff
c:\users\gohlke\appdata\local\temp\pillow-tests\temp_file_libtiff_small_6.tif
--------------------------------------------------------------------
.....................=== remaining temporary files
c:\users\gohlke\appdata\local\temp\pillow-tests\temp_file_libtiff_13.tiff
c:\users\gohlke\appdata\local\temp\pillow-tests\temp_file_libtiff_6.tif
c:\users\gohlke\appdata\local\temp\pillow-tests\temp_file_libtiff_small_13.tiff
c:\users\gohlke\appdata\local\temp\pillow-tests\temp_file_libtiff_small_23.tif
c:\users\gohlke\appdata\local\temp\pillow-tests\temp_file_libtiff_small_30.tiff
c:\users\gohlke\appdata\local\temp\pillow-tests\temp_file_libtiff_small_6.tif
--------------------------------------------------------------------

----------------------------------------------------------------------
Ran 39 tests in 2.859s

OK
running test_file_msp ...
.=== remaining temporary files
c:\users\gohlke\appdata\local\temp\pillow-tests\temp_file_libtiff_13.tiff
c:\users\gohlke\appdata\local\temp\pillow-tests\temp_file_libtiff_6.tif
c:\users\gohlke\appdata\local\temp\pillow-tests\temp_file_libtiff_small_13.tiff
c:\users\gohlke\appdata\local\temp\pillow-tests\temp_file_libtiff_small_23.tif
c:\users\gohlke\appdata\local\temp\pillow-tests\temp_file_libtiff_small_30.tiff
c:\users\gohlke\appdata\local\temp\pillow-tests\temp_file_libtiff_small_6.tif
--------------------------------------------------------------------

----------------------------------------------------------------------
Ran 1 test in 0.032s

OK
running test_file_pcx ...
...=== remaining temporary files
c:\users\gohlke\appdata\local\temp\pillow-tests\temp_file_libtiff_13.tiff
c:\users\gohlke\appdata\local\temp\pillow-tests\temp_file_libtiff_6.tif
c:\users\gohlke\appdata\local\temp\pillow-tests\temp_file_libtiff_small_13.tiff
c:\users\gohlke\appdata\local\temp\pillow-tests\temp_file_libtiff_small_23.tif
c:\users\gohlke\appdata\local\temp\pillow-tests\temp_file_libtiff_small_30.tiff
c:\users\gohlke\appdata\local\temp\pillow-tests\temp_file_libtiff_small_6.tif
--------------------------------------------------------------------

----------------------------------------------------------------------
Ran 3 tests in 0.063s

OK

<snip>

[hugovk]

We don't need run.py now we're using unittest for the tests. I'll remove it along with some other test cleanup I'm in the middle of. I'm also updating the instructions the Tests/README, which will look like:

---

Test scripts are named test_xxx.py and use the unittest module. A base class and helper functions can be found in helper.py.

Run the tests from the root of the Pillow source distribution:

python selftest.py
nosetests Tests/test_*.py

Or with coverage:

coverage run --append --include=PIL/* selftest.py
coverage run --append --include=PIL/* -m nose Tests/test_*.py
coverage report
coverage html
open htmlcov/index.html

To run an individual test:

python Tests/test_image.py

---

Or as a general rule, you can check whatever's in .travis.yml.

Does the problem still occur with nosetests? Remaining temp files should be reported just once by tearDownModule() in helper.py, which is why it's imported into every test.

[hugovk]

The conversion to unittest is "done" but clearly needs more work for Windows...

(The "Not enough memory to handle tile data" warning is also shown on Mac and Travis and was issued but specifically not shown with the old test runner.)

The three errors:

ERROR: large_memory_numpy_test.yield_test
ERROR: large_memory_test.yield_test
ERROR: tester.yield_test

These aren't (and shouldn't be) run by nosetests Tests/test_*.py because large_memory_numpy_test.py and tester.py aren't part of the test set. tester.py has the old helper functions. Running nose on the Tests dir picks up files it thinks are tests (e.g. with test in the filename) but might not be.

Do you have a need for any of the files in Tests/ that are not test_*.py? @wiredfool said at least bench_cffi_access.py, large_*.py and make_hash.py are useful, so we'll keep those, and I'll probably refactor them so we can remove tester.py.

And what do you get from nosetests Tests/test_*.py?

[hugovk]

And there's also python test-installed.py to run against an installed version (for this).

[wiredfool]

Nosetests suppresses stdout by default. That may be where the reports of the files left in the tempdirectory went.

Also, if the test order is rearranged, tearDownModule() can be called multiple times per test file or test run -- it's not a function that's guaranteed to run once at the end of the module.

Take a look at #732. I've changed the way that tempfiles are created in that PR so that they're threadsafe and the reporting shouldn't ever happen more than once for a specific file.

[wiredfool]

This should be fixed via #754, which changed the way that temp files are created and removed.