Improve check_thread_leak output

[gjoseph92]

This should aid in debugging #6796.

This now gives output like:

2 thread(s) were leaked from test

------ Call stack of leaked thread 1/2: <Thread(Thread-3, started 123145522995200)> ------
  File "/Users/gabe/miniconda3/envs/dask-distributed/lib/python3.9/threading.py", line 912, in _bootstrap
        self._bootstrap_inner()
  File "/Users/gabe/miniconda3/envs/dask-distributed/lib/python3.9/threading.py", line 954, in _bootstrap_inner
        self.run()
  File "/Users/gabe/miniconda3/envs/dask-distributed/lib/python3.9/threading.py", line 892, in run
        self._target(*self._args, **self._kwargs)
  File "/Users/gabe/dev/distributed/distributed/tests/test_utils_test.py", line 770, in <lambda>
        t2 = threading.Thread(target=lambda: (event.wait(), "two"))
  File "/Users/gabe/miniconda3/envs/dask-distributed/lib/python3.9/threading.py", line 574, in wait
        signaled = self._cond.wait(timeout)
  File "/Users/gabe/miniconda3/envs/dask-distributed/lib/python3.9/threading.py", line 312, in wait
        waiter.acquire()

------ Call stack of leaked thread 2/2: <Thread(Thread-4, started 123145539784704)> ------
  File "/Users/gabe/miniconda3/envs/dask-distributed/lib/python3.9/threading.py", line 912, in _bootstrap
        self._bootstrap_inner()
  File "/Users/gabe/miniconda3/envs/dask-distributed/lib/python3.9/threading.py", line 954, in _bootstrap_inner
        self.run()
  File "/Users/gabe/miniconda3/envs/dask-distributed/lib/python3.9/threading.py", line 892, in run
        self._target(*self._args, **self._kwargs)
  File "/Users/gabe/dev/distributed/distributed/tests/test_utils_test.py", line 772, in <lambda>
        t3 = threading.Thread(target=lambda: (event.wait(), "three"))
  File "/Users/gabe/miniconda3/envs/dask-distributed/lib/python3.9/threading.py", line 574, in wait
        signaled = self._cond.wait(timeout)
  File "/Users/gabe/miniconda3/envs/dask-distributed/lib/python3.9/threading.py", line 312, in wait
        waiter.acquire()

Instead of:

>               assert False, (bad_thread, call_stacks)
E               AssertionError: (<Thread(asyncio_4, started daemon 123145657024512)>, ['  File "/Users/runner/miniconda3/envs/dask-distributed/lib/pyt...ibuted/lib/python3.10/concurrent/futures/thread.py", line 81, in _worker
E                 \twork_item = work_queue.get(block=True)
E                 '])
E               assert False

which was truncating the traceback and not giving much useful information.

• Tests added / passed
• Passes pre-commit run --all-files

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

       15 files  ±0         15 suites  ±0   6h 35m 10s ⏱️ + 21m 34s
  2 988 tests +1    2 892 ✔️ +2       89 💤 ±0  6 ❌ ±0  1 🔥  - 1 
22 159 runs  +6  21 102 ✔️ +7  1 047 💤 ±0  8 ❌ ±0  2 🔥  - 1 

For more details on these failures and errors, see this check.

Results for commit d1ecee9. ± Comparison against base commit 4b89e26.

♻️ This comment has been updated with latest results.

[jacobtomlinson]

This looks neat! Would be useful to see when this is happening.

[gjoseph92]

• Fix leak in test_localcluster_get_client #6817
• Flaky tests: assert False, (bad_thread, call_stacks) - Worker executor thread still running after Nanny.kill #6796 (what this is meant to help diagnose)