test_stop_single_worker failing on CI

[jakirkham]

Noticed that test_stop_single_worker has started failing on CI. Seems to be a consistent failure. However it wasn't failing on the same code a month ago. So something else has changed (possibly in Distributed) related to how Dask worker directories are managed.

=================================== FAILURES ===================================
___________________________ test_stop_single_worker ____________________________
loop = <tornado.platform.asyncio.AsyncIOLoop object at 0x7f270bb33978>
    def test_stop_single_worker(loop):
        with DRMAACluster(scheduler_port=0) as cluster:
            with Client(cluster, loop=loop) as client:
                cluster.start_workers(2)
                future = client.submit(lambda x: x + 1, 1)
                assert future.result() == 2
                while len(client.ncores()) < 2:
                    sleep(0.1)
    
                a, b = cluster.workers
                local_dir = client.run(lambda dask_worker: dask_worker.local_dir,
                                       workers=[a])[a]
                assert os.path.exists(local_dir)
    
                cluster.stop_workers(a)
                start = time()
                while len(client.ncores()) != 1:
                    sleep(0.2)
                    assert time() < start + 60
>       assert not os.path.exists(local_dir)
E       AssertionError: assert not True
E        +  where True = <function exists at 0x7f27547ba950>('/dask-drmaa/dask-worker-space/worker-c7271lb9')
E        +    where <function exists at 0x7f27547ba950> = <module 'posixpath' from '/opt/anaconda/lib/python3.6/posixpath.py'>.exists
E        +      where <module 'posixpath' from '/opt/anaconda/lib/python3.6/posixpath.py'> = os.path
dask_drmaa/tests/test_core.py:128: AssertionError
=============== 1 failed, 23 passed, 1 xpassed in 120.47 seconds ===============

ref: https://travis-ci.org/dask/dask-drmaa/jobs/396654145#L2188-L2219
ref: https://travis-ci.org/dask/dask-drmaa/builds/385798967

[jakirkham]

Any ideas as to why this failure might have cropped up, @mrocklin?

[mrocklin]

At first glance no, this doesn't seem familiar. Though the timing of this may line up with the release, so I suspect that you're right that it came from upstream.

…

On Mon, Jun 25, 2018 at 8:42 PM, jakirkham ***@***.***> wrote: Any ideas as to why this failure might have cropped up, @mrocklin <https://github.com/mrocklin>? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#93 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszJE8tiC_NUYakgI6Yu8hcatSfvEmks5uAYOCgaJpZM4U3ELq> .

[jakirkham]

Now remembering this test has been historically flaky. ( #60 ) So it may be there is something that dask-drmaa is doing incorrectly that is now being flushed out thanks to a distributed change.

[jakirkham]

Going to mark this as a known failure for now. ( #94 )