Drop to REPL on SIGINT (control-c)

[talolard]

🚀 Feature

During a training run, if the user sends SIGINT (e.g. control-c) drop to a shell (ipython would be amazing)

Motivation

Sometimes I want to stop training right now and do something, probably save an explicitly named checkpoint. I know it's not "the right way" to work, but it is what I (and others?) do or want to do in practice.

As an example, I want to stop training now to try something else, but I want a snapshot of my model at its current state, that is named so I can find it later easily.

Additional context

This would be gross to implement. I imagine it would involve python threads, where a thread is listening for sigint and sets a flag. Then somewhere around on_training_batch_end, if the flag is set we drop to a shell (don't know how)

[carmocca]

This is a cool idea!

If/when #3632 is resurrected, it could be implemented as a callback that registers itself on the ctrl+c signal. But unfortunately, I don't think this will be ready soon

cc: @awaelchli

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!