LBFGS solver part 2: implementing linesearch ensuring sufficient decrease and small curvature #981
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![copybara-service[bot]](https://avatars.githubusercontent.com/in/44061?s=60&v=4){kind=small}
copybara-service
bot
force-pushed
the
test_640585750
branch
7 times, most recently
from
50b624e to
66e60c5
Compare
…ease and small curvature It is recommended to combine LBFGS with a linesearch procedure that ensures not only sufficient decrease but also small curvature along the chosen update direction. We implement the zoom_linesearch presented by Nocedal and Wright, used as reference linesearch method in scipy. The user API follows the one of the scale_by_backtracking_linesearch, i.e., (i) it is a GradientTransformExtraArgs that requires access to value, grad and value_fn, (ii) it can be combined with any base GradientTransform, (iii) value and gradient computed during the linesearch can be recycled using the optax.value_and_grad_from_state utility. PiperOrigin-RevId: 643033634
copybara-service
bot
force-pushed
the
test_640585750
branch
from
66e60c5 to
c4ee3db
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
LBFGS solver part 2: implementing linesearch ensuring sufficient decrease and small curvature
It is recommended to combine LBFGS with a linesearch procedure that ensures not only sufficient decrease but also small curvature along the chosen update direction. We implement the zoom_linesearch presented by Nocedal and Wright, used as reference linesearch method in scipy.
The user API follows the one of the scale_by_backtracking_linesearch, i.e., (i) it is a GradientTransformExtraArgs that requires access to value, grad and value_fn, (ii) it can be combined with any base GradientTransform, (iii) value and gradient computed during the linesearch can be recycled using the optax.value_and_grad_from_state utility.