In multi-trigger, if one trigger crashes, others may keep running

[itowlson]

I created a multi-trigger app with http, timer and sqs triggers. I knew that the SQS trigger was not yet compatible with multi-trigger. The SQS trigger obligingly crashed during startup, but the timer trigger continued to run.

I believe this is due to the timer trigger being a plugin. To run the trigger, we start a spin process, which in turn starts a timer-trigger process. I believe that when we kill that spin process, the child timer-trigger process continues to run.

Further evidence for this is from logging the PIDs of the trigger processes. None of these PIDs matched the ID from doing ps -A | grep timer after SQS had crashed.

This may be a bit gnarly to fix because when we kill the Spin process it just dies, it doesn't get to say "goodnight sweet prince" or (more relevantly) "die, actual plugin process, die" - we can try setting kill_on_drop on the child but I am not sure if finalisers will run in this case. Anyway I will take a look at it and see if I can make a dent.

cc @kate-goldenring if you have any thoughts on this!

[itowlson]

Confirmed finalisers do not appear to run.

[kate-goldenring]

@itowlson I was able to repro this. With a neverends plugin:

#!/bin/bash

i=0
while true; do
   echo $i
   i=$(( i+1 ))
   sleep .5
done

Interestingly adding kill_on_drop(true) did resolve it for me. Stopping the spin (cargo run) process also killed the plugin process

    let mut command = Command::new(binary);
    command.args(args);
    command.envs(get_env_vars_map()?);
    command.kill_on_drop(true);

[itowlson]

Oh interesting! How did you stop the Spin process? In my case the parent process is kill_on_drop-ing the Spin process so maybe your way of stopping is giving it a chance to clean up okay. Thanks!

[itowlson]

I had to add a Ctrl+C handler to external - that seemed to work but more cruft sigh

[kate-goldenring]

I cannot repro my success with my changes, so I agree that kill_on_drop does not work. Will also keep trying!

[itowlson]

AHHH THE CTRL+C HANDLER WORKED AND NOW IT DOESN'T

CURSE YOU ALL

[itowlson]

(not you Kate. just the unfeeling cosmos in general)

[itowlson]

oh SOMETIMES it works