Multithreading stalls if using more threads than cores

[saschatimme]

I noticed the following problem with the first release candiate of 1.2.

julia> versioninfo()
Julia Version 1.2.0-rc1.0
Commit 7097799 (2019-05-30 02:22 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin15.6.0)
  CPU: Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz <---- Quad-Core CPU
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)
Environment:
  JULIA_NUM_THREADS = 4

Assume I have the following snippet:

using HomotopyContinuation, PolynomialTestSystems
f = equations(cyclic(7))
for _ in 1:100
    solve(f)
end

Now assume I start two Julia REPLs where each has Threads.nthreads() ==4.
If I paste the snippet into the first session, everything runs fine, but as soon as I paste the snippet into the second session both sessions freeze, but my CPU load is still 200% in each session.

Note that solve automatically multithreads using the this code (this is basically an disassembled version of the @threads macro). Therefore I think the problem only occurs if there are more threads than cores running.

I think the same problem happens if I execute the test suite of HomotopyContinuation using

pkg> test HomotopyContinuation

At some point the tests just don't continue. If I reduce the test suite to only include the solve(equations(cyclic(7))) tests this problem doesn't happen :(
I know that this is not really an "MWE" but unfortunately I was not able to find a smaller version to trigger this.

[JeffBezanson]

I suspect this might be fixed by #32238.

[JeffBezanson]

I tried this on master but couldn't get it to work due to package version restrictions. Not urgent, but when that is sorted out it will be useful to try this on master. In the meantime I think we can fix it for 1.2.

[saschatimme]

@JeffBezanson The current master of HomotopyContinuation.jl now works again with the Julia master, but I still see the same behaviour on nightly: https://travis-ci.org/JuliaHomotopyContinuation/HomotopyContinuation.jl/jobs/551168625

[JeffBezanson]

Can you reproduce that locally? It works for me.

[saschatimme]

@JeffBezanson Yes, if I run

pkg> add HomotopyContinuation#master
pkg> test HomotopyContinuation

on my local machine the test just get stuck at some point.

julia> versioninfo()
Julia Version 1.3.0-DEV.458
Commit 92f8c50556 (2019-06-27 18:39 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.5.0)
  CPU: Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)
Environment:
  JULIA_NUM_THREADS = 4