Support alternative ways of choosing normal approximations

[sethaxen]

Given an optimization trace, Pathfinder proposes the multivariate normal approximation constructed from the trace that maximizes the ELBO. It does this by approximating the ELBO at each point.

The discussion notes that instead of exhaustively approximating the ELBO at each point, Bayesian optimization could be used to optimize over (or even between) the points. More generally, we could allow alternative objective functions than ELBO and allow any discrete optimizer to be provided. While between points, we could interpolate means, we'd need to think a bit about how to interpolate covariances between points.

[mschauer]

Would this also cover say diagonal approximations to the covariance?

[sethaxen]

In principle, if one could specify a different way of choosing the best distribution, then yes, once could maximize ELBO (or some other objective) over some transformation of that distribution instead.

It might be cleaner though to allow the user to provide such a transformation, which would be applied to all constructed distributions before computing the objective. This would allow the user to control the distributions used for the components of the mixture model returned by multipathfinder.