Keeping only the suspected faulty test (by commenting out rest (!))

alexandrebouchard committed Mar 6, 2023

Removing Turing from test to isolate only one error at the time

alexandrebouchard committed Mar 6, 2023

Check behaviour on different threadlevels

alexandrebouchard committed Mar 6, 2023

Back to same threadlevel as not changing crashing behaviour

alexandrebouchard committed Mar 6, 2023

More conservative tag ub?

alexandrebouchard committed Mar 6, 2023

Revert "More conservative tag ub?" …

This reverts commit 69bb270.

alexandrebouchard committed Mar 6, 2023

Trying on OpenMPI instead of mpich

alexandrebouchard committed Mar 6, 2023

Going back to mpich, trying threadlevel = :multiple

alexandrebouchard committed Mar 7, 2023

Back to funneled as :multiple still crashes

alexandrebouchard committed Mar 7, 2023

Further simplification

alexandrebouchard committed Mar 7, 2023

Candidate fix

alexandrebouchard committed Mar 8, 2023

Reintroducing other tests

alexandrebouchard committed Mar 8, 2023

purge tests from Turing, use only DynamicPPL

miguelbiron committed Mar 8, 2023

Another free

alexandrebouchard committed Mar 8, 2023

Trying to run tests with more rounds

alexandrebouchard committed Mar 8, 2023

system MPI test

miguelbiron committed Mar 9, 2023

fix typos

miguelbiron committed Mar 9, 2023

remove step that does not make sense outside MPI.jl

miguelbiron committed Mar 9, 2023

setup MPIPreferences in the test env

miguelbiron committed Mar 9, 2023

possible fix to Pkg missing

miguelbiron committed Mar 9, 2023

possible fix to Pkg missing

miguelbiron committed Mar 9, 2023

add missing env

miguelbiron committed Mar 9, 2023

another try

miguelbiron committed Mar 9, 2023

test now has its own Project.toml

miguelbiron committed Mar 9, 2023

force Pkg.instantiate in ChildProcess

miguelbiron committed Mar 9, 2023

dont use --project in ChildProcess

miguelbiron committed Mar 9, 2023

ChildProcess inherits the exact same active project

miguelbiron committed Mar 9, 2023

new approach

miguelbiron committed Mar 9, 2023

Changing approach for Isend/isend: explicit Waitall for all requests …

A crash seemed to occur in CI during garbage collection

julia:3364 terminated with signal 11 at PC=7f997aaec971 SP=7f9936ccb970.  Backtrace:
/opt/hostedtoolcache/julia/1.8.5/x64/bin/../lib/julia/libjulia-internal.so.1(ijl_gc_safepoint+0x11)[0x7f997aaec971]

This looks like a crash during garbage collection code (this would be consistent with the fact that it only shows up with allocation-heavy test, i.e. Turing).

So even though the finalizer checks first before calling free, maybe there is something flaky there.

#30 (comment)

alexandrebouchard committed Mar 9, 2023

Is this a Turing multi-threading issue?

alexandrebouchard committed Mar 9, 2023

Testing single-thread for all …

Multi-threading issue seems not specific to Turing as crash occurred
for Blang in last commit.

alexandrebouchard committed Mar 9, 2023

Adding a temporary "dry run" to see if problem cause by some compilation

alexandrebouchard committed Mar 9, 2023

Commenting out Turing and Blang, to see behaviour on toy_mvn and swap… …

…(0.5)

alexandrebouchard committed Mar 9, 2023

openmpi_jll test

miguelbiron committed Mar 9, 2023

openmpi_jll test

miguelbiron committed Mar 9, 2023

openmpi_jll test

miguelbiron committed Mar 9, 2023

openmpi_jll test

miguelbiron committed Mar 9, 2023

more messages from Child processes

miguelbiron committed Mar 9, 2023

extra arg to OpenMPI

miguelbiron committed Mar 9, 2023

rm mpi version query

miguelbiron committed Mar 9, 2023

use old julia_cmd + rm manifest as in MPI.jl CI

miguelbiron committed Mar 9, 2023

add --oversubscribe for OpenMPI + rethrow exception

miguelbiron committed Mar 9, 2023

simplify logging

miguelbiron committed Mar 9, 2023

force instantiate + precompile

miguelbiron committed Mar 9, 2023

Add mpi_args to mpi_test

alexandrebouchard committed Mar 10, 2023

Temporary: trying to speed up some key tests

alexandrebouchard committed Mar 10, 2023

Fix

alexandrebouchard committed Mar 10, 2023

Fix the fix + reintroducing the system-MPI tests

alexandrebouchard committed Mar 10, 2023

Add back libmpich-dev to resume investigation on ghostbug

alexandrebouchard committed Mar 10, 2023

Trying to simplify CI setup needed to reproduce ghostbug

alexandrebouchard committed Mar 10, 2023

Fix last commit

alexandrebouchard committed Mar 10, 2023

toy_mvn not enough to manifest ghostbug, trying Turing

alexandrebouchard committed Mar 10, 2023

test mpich+openmpi using brew

miguelbiron committed Mar 10, 2023

fix wrong abi detection for mpich

miguelbiron committed Mar 10, 2023

move xtra args to MPI struct + remove prints + failsafe for empty cur… …

…rent project

miguelbiron committed Mar 10, 2023

mpiexec args for childprocess

miguelbiron committed Mar 10, 2023

re-introduce all CI tests + fix bug in building mpi cmd

miguelbiron committed Mar 11, 2023

add support for using without Project.toml

miguelbiron committed Mar 11, 2023

add comment explaining why we wait on Isend

miguelbiron committed Mar 11, 2023

mpiexec_args is a Cmd now

miguelbiron committed Mar 11, 2023

re-instate all tests

miguelbiron committed Mar 11, 2023

Test hypothesis that GC+multithread is issue; determine all MPIs affects

alexandrebouchard committed Mar 11, 2023

force instantiate in mpi_test + add MicrosoftMPI test

miguelbiron committed Mar 11, 2023

adding back all test

miguelbiron committed Mar 12, 2023

Change mpi_active impl to fix open mpi bug …

Before this, we had crashed such as in
18ef655

Here is what was happening in these crash
1. parent process is using a system library
2. parent process' test first call local pigeons
3. that in turns called mpi_active() which internally used MPI.Init()
   to see if Comm_size > 1
4. that had the side effect of changing ENV
5. then, when calling pigeons MPI tests, these ENV variables were
   passed to ChildProcess, causing problems

The new approach avoids to call MPI.Init() frivolously

alexandrebouchard committed Mar 13, 2023