jl_current_exception doesn't return the exception when called from atexit

[simonbyrne]

In Julia 1.8 (and earlier):

├ julia -e 'atexit(() -> @show ccall(:jl_current_exception, Any, ())); error("oh no")'
ERROR: oh no
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] top-level scope
   @ none:1
ccall(:jl_current_exception, Any, ()) = ErrorException("oh no")

On the current nightly:

├ ./julia --startup-file=no -e 'atexit(() -> @show ccall(:jl_current_exception, Any, ())); error("oh no")'
ERROR: oh no
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] top-level scope
   @ none:1
ccall(:jl_current_exception, Any, ()) = nothing

There is similar behavior with current_exceptions() as well. This is a problem as there isn't a good way to determine if we're exiting due to an error or not (#35981)

[gbaraldi]

Bisected to e34860e

[simonbyrne]

@vtjnash any suggestions for what we can do here? We don't need to recreate the old behavior, just need some way to find the error stack when we exit due to an error.

[JeffBezanson]

How do MPI bindings for other languages handle this?

[simonbyrne]

The official C and Fortran ones require you to manually call MPI_Finalize, or call MPI_Abort on an error.

mpi4py, which is probably the most widely used 3rd party one, and the closest to MPI.jl, by default adds an atexit hook that calls MPI_Finalize:
https://github.com/mpi4py/mpi4py/blob/bd9e2a58be3689983f9fff798447a76796b10e5a/src/mpi4py/MPI/atimport.pxi#L328-L334
It also provides a wrapper module
https://mpi4py.readthedocs.io/en/stable/mpi4py.run.html
that will check catch any errors and trigger an MPI_Abort if necessary
https://github.com/mpi4py/mpi4py/blob/b7ca61d43aec14d23d64490d2b8591974cb81d4e/src/mpi4py/run.py#L188

[JeffBezanson]

Ok, so mpi4py is only able to use MPI_Abort if you go through that wrapper, but we want it to work without such a wrapper?

[simonbyrne]

Ideally, yes.