bpo-31783: Fix a race condition creating workers during shutdown

[brianquinlan]

https://bugs.python.org/issue31783

[markshannon]

Can you add a test for this, thanks.

[brianquinlan]

The race condition occurs with the following set of context switches:

Main Thread                           Thread  #1
========= .                         ========
1.                                   ThreadPoolExecutor.submit()
                                        ...
                                        if _shutdown:

                    <=====

2. thread._python_exit
      _shutdown = true
      items = list(_threads_queues.items())

                    =====>

3.                                      ...
                                         _threads_queues[t] = self._work_queue

                    <=====

4. 
    for t, q in items:
        q.put(None)
    for t, q in items:
        t.join()

At this point there is a new thread added to _threads_queues but thread._python_exit won't join it because it was added to _threads_queues after a copy was made (i.e.
items = list(_threads_queues.items()))

The bug has a patch with time.sleep() to force the pathological sequencing and a test script to demonstrate the problem.

[brianquinlan]

I don't think writing a test for this would be feasible because provoking the race condition requires very specific timing that, at least on my machine, I can't provoke without adding sleep calls to the code.

[bedevere-bot]

@brianquinlan: Please replace # with GH- in the commit message next time. Thanks!

[miss-islington]

Thanks @brianquinlan for the PR 🌮🎉.. I'm working now to backport this PR to: 3.7.
🐍🍒⛏🤖

[miss-islington]

Thanks @brianquinlan for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

[miss-islington]

Sorry, @brianquinlan, I could not cleanly backport this to 3.8 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 242c26f53edb965e9808dd918089e664c0223407 3.8

[miss-islington]

Sorry @brianquinlan, I had trouble checking out the 3.7 backport branch.
Please backport using cherry_picker on command line.
cherry_picker 242c26f53edb965e9808dd918089e664c0223407 3.7