Making auto-sklearn fit methods be interactively stoppable and still give a fitted model

[seewalker]

Recently I was using auto-sklearn and interrupted the fitting with control-C. It occurred to me that if I'd specified the elapsed time so far up front as the value of the time_left_for_this_task l would have a trained model. While it is good that auto-sklearn can have the metalearners be immediately interruptible, it would also be nice to have them be interruptible in a way that still gives results.

More concretely, maybe the program could listen for signals from the OS and respond to "control-C" with an immediate exit and "control-D" with a answer-preserving eventual exit. This would be a good answer for linux and OSX at least which has a concept of signals, I'm not sure about microsoft. For the fully platform independent case maybe automl could listen for commands typed from stdin, like "break" or "soft_stop" could cause a answer-preserving exit and "exit" or "hard_stop" could cause an immediate exit. Does automl already use stdin for anything of importance while fit is running? If so, that might not be a good idea.

Does this sound like a welcome idea to project maintainers? If so, I'd be happy to be the one to implement it. I did raise a few questions about what interface/commands would be best for distinguishing interactive exits, and maybe project maintainers have some judgement about good decisions to mention to me (if I'm going to be the one to do this if it is to happen).

[mfeurer]

Thanks for your interest in Auto-sklearn @seewalker. You're right that there are use cases where interrupting Auto-sklearn and getting a trained model is desirable. We're mostly working in a batch setting, so we never ran into this issue. We'd definitely be interested in this if we find a nice way of integrating this feature. Let me give you a brief overview of what to keep in mind for this to work:

• we don't need to care about Windows as we're not supporting it in the first place.
• we currently hand control over to a program called SMAC to do the optimization. We could simply start this in a new process and listen to any signals/inputs in the main process.
• I think control-C is a good signal to actually stop training - however, a user would have to press control-C twice in order to get the common behavior of a shell program - namely that it exits. One would probably have to emit a warning in this case.

You would have to start from the development branch and basically need to start around line 400 of the file automl.py. Let me know if you have any further questions.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs for the next 7 days. Thank you for your contributions.