Improve handling of failed evaluations #154
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…g/optimas into feature/failed_trials
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
…g/optimas into feature/failed_trials
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
…g/optimas into feature/failed_trials
for more information, see https://pre-commit.ci
AngelFP
changed the title
for more information, see https://pre-commit.ci
…g/optimas into feature/failed_trials
|
Thanks for this PR @AngelFP. Could you fix the conflicts with the |
Good you noticed that. Conflicts solved :) |
RemiLehe
reviewed
Feb 12, 2024
| # is changed to the exploration directory after the call to `libE`. | ||
| # As a workaround, the cwd is stored and then set again at the end of | ||
| # `run`. | ||
| cwd = os.getcwd() |
There was a problem hiding this comment.
Should we bring this up to the libEnsemble team?
There was a problem hiding this comment.
Good point. I opened a new issue Libensemble/libensemble#1244
RemiLehe
approved these changes
Feb 12, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Addresses #143, #144.
This PR adds a status parameter to
Trial, which can be eitherCANDIDATE,RUNNING,COMPLETEDorFAILED. This status is also used to informAxof whether a trial has failed, so that it can be properly handled by the surrogate model. Failed trials in the Ax Service generators can be labeled asFAILEDorABANDONED, where the latter implies that the failed trial will not be suggested again. This behavior is controlled by the new parameterabandon_failed_trials(Trueby default).With the proposed implementation, trials will be considered as failed if any of these two conditions are met:
TemplateEvaluator.NaNfor the value of any of the objectives. This applies to all evaluators.Case 2 includes the case in which the evaluation or analysis function failed to provide a value of the objective. Previously, this would result in the objective being returned with
0as value, which could confuse the optimizer.Changes
TrialStatusclass.statusproperty toTrialand other related methods.trial_statustohistory.NaNs.FAILEDtrials in the Ax generators. By default they are set asABANDONEDso that they are not suggested again. This behavior can be controlled with theabandon_failed_trialsargument.sim_specs["out"].completedandevaluatedtrials. Anevaluatedtrial is one whose evaluation has completed or failed.cwdto change when running with threading comms.