Disables siginterrupt for SIGUSR1 #1844
Conversation
I've recently started using SIGUSR1 to stop old workers on deploy and each deploy has resulted in all MapReduce tasks dying with `IOError: [Errno 4] Interrupted system call`. Setting siginterrupt to False for SIGUSR1 prevents this error for me.
|
Looks good to me. The concept of |
|
Siginterrupt is new to me as well, but I was able to solve the problem thanks to some Googling. I can't find the Stack Overflow thread that helped me solve it again though :(. The system call that got interrupted most of the time was reading the stderr pipe in the spawned subprocess for the mapreduce on line 295 of luigi/contrib/hadoop.py. https://github.com/spotify/luigi/blob/master/luigi/contrib/hadoop.py#L295 If the signal happened during scheduling, it could kill the multiprocessing queue get during a call to conn.recv() which I'm guessing is also a pipe read. |
Description
Sets siginterrupt to False for SIGUSR1 when installing the shutdown handler.
Motivation and Context
I've recently started using SIGUSR1 to stop old workers on deploy and each deploy has resulted in all MapReduce tasks dying with
IOError: [Errno 4] Interrupted system call. To avoid this, we want the SIGUSR1 signal to not interrupt system calls.Have you tested this? If so, how?
I wrote some unit tests and tested it heavily in production both by sending lots of SIGUSR1 signals to running MRs and by seeing the regular stream of errors during deploys stop.