Graceful warning when trying to kill a worker that's running a stage

[rachitnigam]

Trap the first Ctrl-C to workproc and show a warning if the worker is running a job.

On the second Ctrl-C, gracefully execute a cleanup phase to write back changes from the make stage.

[rachitnigam]

@sampsyo any comments on how to propogate exceptions into a worker? Events seem to be the canonical way to do this in python but not sure if they can interrupt a running make job.

[sampsyo]

Hmm, not entirely sure what you mean. You’re trying to trap a ^C in one thread and signal another thread to handle it? Or are you trying to handle other exceptions and display them on the main thread?

I don’t know what an Event is, but I think you’d use a standard synchronization construct like a condition variable or a concurrent queue.

[rachitnigam]

I want do the former where Ctrl-C in the main thread informs workers to release ongoing jobs (mark them as failed or revert them to the original state) and cleanup job files.

Right now, if a worker is killed while executing a fpga_execute stage or make stage, it leaves the job in the locked state.

[sampsyo]

Aha! It’s a little tricky because you’d need to interrupt a running subprocess (e.g., a make invocation). Needless to say, the worker thread would need to be available (not blocked waiting for the process to complete) to receive the message so it can send a Unix signal, like SIGINT or SIGKILL, to the subprocess. That’s possible, but it would require a redesign of the way the worker invokes subprocesses—it currently launches the subprocess and then blocks until it completes or times out, which won’t do.

Alternatively, you could possibly imagine leaving the process hanging out there but just changing the status. In that case, you’d probably want to have the stage threads proactively inform the main thread which jobs they’re working on, and then the master can make state changes itself when it dies.

Concurrency is hard!