Random exception in python when using multiprocessing due to rdseed failure

[alexisshaw]

When I am running a large number of samples simulations simultaneously using stim in a massively multi-process environment (python not supporting multi-threading, I use multi-process computation using python multithreading starmap), this happens semi-randomly when I run a multi-threaded simulation, see the stack trace below:

This seems to be related to a multi-threading bug in libstdc++, so perhaps using the approach used by the google cloud c++ api would be a good approach. See:

googleapis/google-cloud-cpp-common#208

googleapis/google-cloud-cpp-common#272

seeing as the call to random_device is only done once per process in PYBIND_SHARED_RNG(), it is probably safe performance-wise to use the second fix, as the performance regression should be very minor.

This seems to be related to the shared rdseed buffer in certain Intel cpus.

[Strilanc]

The fix you're suggesting is

#if defined(__linux) && defined(__GLIBCXX__) && __GLIBCXX__ >= 20200128
    std::random_device rd("/dev/urandom");
#else
  std::random_device rd;
#endif 

instead of just std::random_device rd;?

[Strilanc]

Urgh, looking over some of the threads on this around the internet. what a gigantic mess. There's something deeply wrong when you have to workaround standard library functionality not doing what it's supposed to do.

[alexisshaw]

Yeah, basically, for the immediate fix.

And don't I know it. This really should be handled by the standard library, why it isn't is super annoying.

[alexisshaw]

I mean technically it is allowed to except in the standard, but the failure mode really should be more graceful here.

[Strilanc]

@alexisshaw Could you try pip installing this file into the relevant environment, and confirming that it makes the problem go away?

stim-1.3.dev0.tar.gz

[Strilanc]

I'm going to move forward with publishing a release with this fix to avoid holding up other bug fixes.