Gradient optimizers

[demroz]

adds a few gradient optimizers

• gradient descent
• nesterov
• and L-BFGS

everything is policy based, so these should be quite extensible. I may add a few more optimziers in the future, but these are the main ones i think. If there are any specific ones that you guys think should be added, I'd be happy to do so. Also although everything is reverse-mode autodiff centric, as long as you provide the objective function, a way to evaluate it, and a way to evaluate the gradient, everything should work correctly.

For some examples on how to use the optimziers: test_gradient_descent_optimizer.cpp, test_nesterov_optimizer.cpp, and test_lbfgs.cpp should be good starting points.

I'm working on the documentation currently. I wanted to hold off to see if any major revisions are necessary

[codecov]

Codecov Report

❌ Patch coverage is 97.74011% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.24%. Comparing base (2697069) to head (9e78468).
⚠️ Report is 11 commits behind head on develop.

Files with missing lines	Patch %	Lines
...detail/reverse_mode_autodiff_memory_management.hpp	95.91%	4 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #1328   +/-   ##
========================================
  Coverage    95.24%   95.24%           
========================================
  Files          814      814           
  Lines        69309    69312    +3     
========================================
+ Hits         66015    66019    +4     
+ Misses        3294     3293    -1     

Files with missing lines	Coverage Δ
...de/boost/math/differentiation/autodiff_reverse.hpp	98.78% <ø> (ø)
test/test_functions_for_optimization.hpp	100.00% <100.00%> (ø)
...st_reverse_mode_autodiff_flat_linear_allocator.cpp	100.00% <100.00%> (ø)
...detail/reverse_mode_autodiff_memory_management.hpp	95.91% <95.91%> (+1.02%)	⬆️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2697069...9e78468. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.