Print threads on stuck on flaky test

[marcotc]

This PR is meant to help debug flaky test runs that get stuck inside the execution of the #work_pending? test.

Here's an example where CircleCI shuts down a container with no output after 10 minutes of inactivity:

Datadog::Core::Workers::IntervalLoop
  when included into a worker
    #loop_wait_time=
      is expected to change `worker.loop_wait_time` from 1 to 0.6986260331927575
    #work_pending?
      when the worker is not running
        is expected to equal false
      when the worker is running
Killed
/usr/local/bin/ruby -I/usr/local/bundle/gems/rspec-core-3.12.0/lib:/usr/local/bundle/gems/rspec-support-3.12.0/lib /usr/local/bundle/gems/rspec-core-3.12.0/exe/rspec --pattern spec/\*\*/\*_spec.rb  --exclude-pattern spec/\*\*/\{contrib,benchmark,redis,opentracer,auto_instrument,opentelemetry\}/\*\*/\*_spec.rb,\ spec/\*\*/\{auto_instrument,opentelemetry\}_spec.rb failed
rake aborted!
Command failed with status (1): [bundle exec appraisal ruby-2.7.6-core-old ...]

[codecov-commenter]

Codecov Report

Merging #2636 (f62009c) into master (f158b71) will decrease coverage by 0.03%.
The diff coverage is 46.66%.

@@            Coverage Diff             @@
##           master    #2636      +/-   ##
==========================================
- Coverage   98.07%   98.04%   -0.03%     
==========================================
  Files        1153     1153              
  Lines       63128    63158      +30     
  Branches     2813     2817       +4     
==========================================
+ Hits        61912    61925      +13     
- Misses       1216     1233      +17     

Impacted Files	Coverage Δ
spec/datadog/core/workers/interval_loop_spec.rb	89.40% <46.66%> (-10.60%)	⬇️
...ec/datadog/tracing/contrib/sidekiq/patcher_spec.rb	96.00% <0.00%> (-4.00%)	⬇️
...atadog/tracing/contrib/grpc/support/grpc_helper.rb	98.24% <0.00%> (-1.76%)	⬇️
lib/datadog/core/diagnostics/environment_logger.rb	97.69% <0.00%> (-0.77%)	⬇️
...dog/profiling/collectors/cpu_and_wall_time_spec.rb	97.98% <0.00%> (+0.40%)	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[ivoanjo]

Soo... I strongly suspect hang in IntervalLoop is #2271 (comment) .

I am somewhat wary of creating a new thread for every testcase to debug IntervalLoop; can I convince you to reduce the scope of this PR just to that module's specs, and not to the whole test suite?

[marcotc]

Sure, will reduce the scope.

[marcotc]

Soo... I strongly suspect hang in IntervalLoop is #2271 (comment) .

The case where it just completely hangs has been around for quite a bit, but I've yet to be able to reproduce it.

[marcotc]

There are two tests that sometimes flake in the same file, moved to those.

[marcotc]

Hey, it happened in this branch, but it didn't print anything :(
https://app.circleci.com/pipelines/github/DataDog/dd-trace-rb/9046/workflows/6dea0b42-9df5-4ba9-82d2-84c764637028/jobs/336536
I have to investigate it.

[ivoanjo]

I guess it may help to sprinkle a few sleeps and Thread.pass on the involved methods to cause some less-frequent interleavings.