exit() after doing an rmprocs() takes almost 60 seconds to exit.

[amitmurthy]

The following :

julia> addprocs(1)
1-element Any Array:
 2

julia> rmprocs(workers())

julia> Worker 2 terminated.


julia> exit()

results in exit() taking 60 seconds to return.

strace -p on the julia process shows a continuous call to epoll for 60 seconds.

...
epoll_wait(5, {}, 1024, 100)            = 0
epoll_wait(5, {}, 1024, 0)              = 0
epoll_wait(5, {}, 1024, 101)            = 0
epoll_wait(5, {}, 1024, 0)              = 0
epoll_wait(5, {}, 1024, 101)            = 0
...

[amitmurthy]

Without a call to rmprocs() it is instantaneous

[Keno]

Did you terminate all the sockets correctly?

[amitmurthy]

No, the sockets are not being closed due to #3495

I'll leave this open till that is fixed and then retest with a stop_reading.

But shouldn't exit() just clean up all resources and exit? What is it waiting on?

[Keno]

It's trying to cleanly shutdown everything, so you don't lose data. Probably still a bug that it takes so long though. Try using close to close the sockets before.

[amitmurthy]

For this particular socket, the other end has already closed the connection - the worker process has exited.

[vtjnash]

this is the fault of the finalizer for a RemoteRef calling send_del_client calling worker_from_id. However, the worker has been deleted, so it goes to sleep for 60 seconds before exiting:

function worker_from_id(pg::ProcessGroup, i)
#   Processes with pids > ours, have to connect to us. May not have happened. Wait for some time.
    start = time()
    while (!haskey(map_pid_wrkr, i) && ((time() - start) < 60.0))
        sleep(0.1)      
        yield()
    end
    map_pid_wrkr[i]
end

[amitmurthy]

While the above should be fixed, I think it makes sense to have

1. exit() wait for a maximum of 10 seconds (arbitrary, I know) for a clean shutdown and it it has not happened, just go ahead and ccall system exit function.

2. A variant of exit(code::Integer, now=false) that just does a ccall to system exit, if now = true

[StefanKarpinski]

Why not make the time to wait the option? And zero means exit right away, defaults to ten.

[amitmurthy]

Good idea. I agree.

Since jl_exit has already started the shutdown process, I guess we cannot use any of the regular calls to setup a Julia timer...

Can the implementation of this use SIGALRM - set using the alarm system call - for the process to be terminated after time seconds?

Windows will need an equivalent to SIGALRM

Or can libuv provide any support for this?

[amitmurthy]

Should I file the requirement for an exit with a wait time as a separate issue? I think it is a valid requirement since the finalizers in user code can also cause delays for a clean exit.

[vtjnash]

no. it is an awful idea to allow a finalizer to sleep. this could break your code in any number of ways. I will try to push a fix that turns calls from finalizers to yield into errors