Handle deserialization errors in RemoteExceptions

[amitmurthy]

If an exception type (or any data fields in the exception) thrown in a worker is not present on the destination node, deserialization of the exception results in a loss of the original error and the stacktrace (if the original exception is wrapped in a CapturedException).

The root cause is shown below:

julia> remotecall_fetch(()->eval(quote
         type Foo <: Exception
           x
         end
         throw(Foo(1))
       end), 2)
ERROR: UndefVarError: Foo not defined
 in deserialize_datatype(::Base.ClusterSerializer{TCPSocket}) at ./serialize.jl:841
 in handle_deserialize(::Base.ClusterSerializer{TCPSocket}, ::Int32) at ./serialize.jl:580
 in deserialize(::Base.ClusterSerializer{TCPSocket}, ::DataType) at ./serialize.jl:909
 in deserialize_datatype(::Base.ClusterSerializer{TCPSocket}) at ./serialize.jl:856
 in handle_deserialize(::Base.ClusterSerializer{TCPSocket}, ::Int32) at ./serialize.jl:580
 in deserialize(::Base.ClusterSerializer{TCPSocket}, ::DataType) at ./serialize.jl:909
 in deserialize_datatype(::Base.ClusterSerializer{TCPSocket}) at ./serialize.jl:856
 in handle_deserialize(::Base.ClusterSerializer{TCPSocket}, ::Int32) at ./serialize.jl:580
 in deserialize_msg(::Base.ClusterSerializer{TCPSocket}, ::Type{Base.ResultMsg}) at ./multi.jl:120
 in deserialize_msg(::Base.ClusterSerializer{TCPSocket}) at ./multi.jl:130
 in message_handler_loop(::TCPSocket, ::TCPSocket, ::Bool) at ./multi.jl:1371
 in process_tcp_streams(::TCPSocket, ::TCPSocket, ::Bool) at ./multi.jl:1328
 in (::Base.##505#506{TCPSocket,TCPSocket,Bool})() at ./event.jl:73
Stacktrace:
 [1] #remotecall_fetch#493(::Array{Any,1}, ::Function, ::Function, ::Base.Worker) at ./multi.jl:1093
 [2] remotecall_fetch(::Function, ::Base.Worker) at ./multi.jl:1085
 [3] #remotecall_fetch#496(::Array{Any,1}, ::Function, ::Function, ::Int64) at ./multi.jl:1106
 [4] remotecall_fetch(::Function, ::Int64) at ./multi.jl:1106

While it is difficult to recover a valid object after a deserialization error, we could at least capture the type of the original error, say a BoundsError as part of the deserialization error stack.

Since remotecalls always wrap the remote exception in a RemoteException we can capture a bit more information of the original error. We could

• add a short stringified description of the original error type and the first line from the original callstack as the first field to be deserialized in a specific deserialize(s::AbstractSerializer, t::Type{T}) where T <: RemoteException implementation.
• This implementation will catch any further deserialization errors and rethrow it with the short stringified desc of the original error added.

This way some information of the original error is retained even in the event of deserialization errors.

Ref : #20027 (comment)