Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unconditionally cleanup temp folder in poolRunner #173

Merged
merged 2 commits into from
Nov 15, 2017
Merged

Unconditionally cleanup temp folder in poolRunner #173

merged 2 commits into from
Nov 15, 2017

Conversation

althonos
Copy link
Contributor

Hi Nathan,
because of a conditional branch the temporary directories that a poolRunner call creates are not properly created in Python 3. When running tests several times, the /tmp (or equivalent) directory can become cluttered very quickly. I removed the if branching so that the temporary folder is always removed.

@coveralls
Copy link

coveralls commented Nov 13, 2017

Coverage Status

Coverage remained the same at 100.0% when pulling 6ffde65 on althonos:patch-1 into d260d27 on CleanCut:master.

@coveralls
Copy link

coveralls commented Nov 15, 2017

Coverage Status

Coverage remained the same at 100.0% when pulling 185b526 on althonos:patch-1 into d260d27 on CleanCut:master.

1 similar comment
@coveralls
Copy link

Coverage Status

Coverage remained the same at 100.0% when pulling 185b526 on althonos:patch-1 into d260d27 on CleanCut:master.

@CleanCut
Copy link
Owner

CleanCut commented Nov 15, 2017

Unfortunately, that (somewhat perplexingly) breaks skipping tests from setUpClass class methods on Python 3 on macOS.

It consistently fails on both Travis and on my MacBook Pro.

I think the fact that I couldn't figure out why it broke it is why that seemingly-silly if statement was there in the first place.

Lets take another look and see if we can fix the actual problem, instead of leaving temp files around.

venv3.6/bin/python -m green.cmdline -s 0 -k green
...................................................................................................s.....................................
...................................................E..........sTraceback (most recent call last):
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/util.py", line 262, in _run_finalizers
    finalizer()
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/util.py", line 186, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/Users/nathan/proj/green/venv3.6/lib/python3.6/shutil.py", line 471, in rmtree
    onerror(os.lstat, path, sys.exc_info())
  File "/Users/nathan/proj/green/venv3.6/lib/python3.6/shutil.py", line 469, in rmtree
    orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/8y/cgqfhxyn2fz3r8n627_6dm_m0000gn/T/tmpdc5m6xlx/pymp-ljgmarin'


Error in green.test.test_suite.TestFunctional.test_skip_in_setUpClass
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/case.py", line 605, in run
    testMethod()
  File "/Users/nathan/proj/green/green/test/test_suite.py", line 158, in test_skip_in_setUpClass
    result = run(tests, self.stream, self.args)
  File "/Users/nathan/proj/green/green/runner.py", line 91, in run
    manager = multiprocessing.Manager()
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/context.py", line 56, in Manager
    m.start()
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/managers.py", line 517, in start
    self._address = reader.recv()
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError

Ran 200 tests in 3.258s

FAILED (errors=1, passes=197, skips=2)

@althonos
Copy link
Contributor Author

@CleanCut : I can't help you with that, I have no way to test on MacOS. What about reverting to

if sys.platform != 'darwin' or sys.version_info[0] == 2: # pragma: no cover
    shutil.rmtree(tempfile.tempdir, ignore_errors=True)

as a temporary patch ?

@CleanCut
Copy link
Owner

@althonos Sure, lets do that. I'd prefer actually fixing it for reals somehow, but I'm swamped right now. Lets add a # TODO: Make this not necessary on macOS+Python 3 so we can actually clean up after ourselves

@coveralls
Copy link

coveralls commented Nov 15, 2017

Coverage Status

Coverage remained the same at 100.0% when pulling 653dd43 on althonos:patch-1 into d260d27 on CleanCut:master.

1 similar comment
@coveralls
Copy link

Coverage Status

Coverage remained the same at 100.0% when pulling 653dd43 on althonos:patch-1 into d260d27 on CleanCut:master.

@CleanCut CleanCut merged commit 2daf8d9 into CleanCut:master Nov 15, 2017
@CleanCut
Copy link
Owner

Included in 2.11.2, just released.

@CleanCut
Copy link
Owner

I had to revert this commit in 2.13.1, as it was breaking Green on some setups. Do you want to look at this again with 2.13.1 and see if you still see a buildup of extra temporary directories?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants