Strange race condition while running conjugate gradients in multithreaded code

[ranjanan]

I've been trying to debug the following race condition for a while now with little success, and I'm posting here just in case this unconvers a bug in the threading runtime.

Here's how to reproduce it on Julia 1.4 (same behaviour on master):

1. Add the package Circuitscape with the branch "RA/mt"
2. Start Julia with multiple threads

Then run:

foo = Vector{Any}()
buzz = Ref{Any}()
using Base.Threads
@show nthreads() # shows threads
using Circuitscape
cd("~/.julia/dev/Circuitscape/test")
compute(test_problem("sgVerify1.ini"))

Now, if you enter fetch.(buzz[]) at the REPL, you should see a few vectors, some of which have NaN's in them, which is incorrect. If you do the above steps but with single threaded julia, you should see proper floating point values, which are expected.

I've tried to obtain an rr recording as well, but for some puzzling reason, I cannot reproduce this race condition using rr. The command I've been running:
JULIA_NUM_THREADS=4 /opt/rr/bin/rr record --chaos --num-cores=4 /data/ranjanan/julia/usr/bin/julia-debug

Any help with this to understand the underlying issue would be greatly appreciated.

cc: @Keno @vchuravy

[Keno]

typeof(cmap) = [ Info: 2020-06-15 15:49:29 : Time taken to write voltage maps = 0.000110052 seconds
type GraphData has no field cumArray{Float64,2}
Stacktrace:

Can you provide an exact Project/manifest?

[ranjanan]

Yes, that error is expected. You should still be able to look at fetch.(buzz[]) and see NaN's.
Project.toml:

[deps]
Circuitscape = "2b7a1792-8151-5239-925d-e2b8fdfa3201"

Manifest:
https://pastebin.com/FhD4wnB7

[Keno]

Ok, I can reproduce this. Looks like a data race. I'm thinking ThreadSanitizer might be the next step.

[ViralBShah]

@ranjanan Can you point to where the threaded part of the code is?

[antoine-levitt]

Does this use FFTs? JuliaMath/FFTW.jl#141 and comments in it might be relevant

[ViralBShah]

No FFTs in here. It's all sparse matrix stuff.

[ranjanan]

The multi-threaded code in question is here.

[ViralBShah]

@Keno Is this an issue in the user code or an issue in our multi-threading system? If @ranjanan has to use ThreadSanitizer, is there any documentation on how one might go about doing it?

[Keno]

Probably an issue in the user code. ThreadSanitizer can be turned on in Make.user (see here: #36929). Note though that any external libraries that could be participating in the data race need to be built with tsan enabled also, which we currently don't really have a good way to do.

[ViralBShah]

This codebase is primarily native Julia stuff.

[ViralBShah]

I believe we debugged this to a race condition in the user code in AlgebraicMultigrid.jl.