Options for approximation?

[cscherrer]

Since the MAP step seems to run to convergence anyway, it seems like it might be straightforward to get to a Laplace approximation. In some settings, it could be useful to have this instead of or in addition to the variational fit. Maybe this could be a configuration option?

[sethaxen]

Pathfinder fits a sequence of multivariate normal approximations to the target distribution. When the optimization trajectory converges to a MAP estimate (not guaranteed or enforced), then the final distribution in this sequence will approximate the Laplace approximation, where the closeness depends on the accuracy of the estimate of the inverse Hessian constructed from the history of the optimization trajectory. Of course, not all distributions have a MAP, and even if the optimization converges to a mode, that doesn't mean it's the MAP. This is why returning the ELBO-maximizing distribution in the sequence is important and useful for more distributions.

In the next release, the entire sequence of distributions will be exposed to the user, so they are free to just grab the last one and use it like a Laplace if they like (#19). I don't think we want to support constructing the exact Laplace distribution (i.e. computing the exact Hessian). This can be done in ~2 lines of code without Pathfinder, so it doesn't seem to be worth the extra complexity.

But we may want to support different optimizations than maximizing the ELBO (#15), and if we have an interface for that, it would be easy to provide one that just always chooses the last distribution as the one to be returned. But this would be wasteful compared to just 1) performing MAP and 2) computing the Hessian (or an approximation to it)

[cscherrer]

Great point, just having access to the intermediate computations would be enough, and this package can stay light weight. I'll close this issue, let me know or reopen if you think there's more to discuss.