-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add option to stop search once a solution is found #13
Add option to stop search once a solution is found #13
Conversation
@microsoft-github-policy-service agree |
Can you elaborate on this usecase? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
For example, if you wanted to reproduce the experiment in the retro* / PDVN paper, or make a box plot of the time at which the first solution is found, this only requires finding on solution (that is all they report). The students I am working with are trying to evaluate value functions in terms of how quickly they guide a search algorithm to a solution. That is another use case. Does that make sense? |
Co-authored-by: Krzysztof Maziarz <krzysztof.maziarz@microsoft.com>
In #13, a `stop_on_first_solution` flag was added to the base search algorithm class, but so far it was not available when using the search CLI. As this option can be quite useful (for e.g. running a quick check to assess the difficulty of a given set of targets), this PR adds it as a CLI option as well.
Rationale: Even though our main use case for
syntheseus
so far has been to run search for a fixed amount of time and analyze all routes in the resulting search graph, there are some scenarios where we only care about whether or not a route is found (e.g. benchmarking). This is pretty simple to check so I thought it is worth supporting.Implementation: previously all algorithms called
time_limit_reached
each iteration to decide whether to stop early. I just augmented this function to also check for stopping once a solution is found. This involved:stop_on_first_solution
kwarg to the base algorithm class (defaults to False, which is the most sensible default in my opinion)time_limit_reached
which checksself.stop_on_first_solution and graph.root_node.has_solution
. This required adding agraph
argument to the function (previously it took no arguments), causing changes in all algorithm files.time_limit_reached
no longer seemed like a descriptive name for the function, so I changed it toshould_stop_search
syntheseus/tests/search/algorithms/test_base.py
which tests the effectiveness of this kwarg for all algorithms (because algorithm-specific tests all subclass the base test)Rejected alternative: although it would be appealing to generalize this to "stop after n solutions" or "stop after n diverse solutions" this is not simple or quick to check so I don't think we should support it out-of-the-box. If users want this then they could implement it themselves in a subclass.