[WIP] Riemannian Manifold HMC

[bhargavvader]

PR to discuss and implement RM-HMC.
Right now I'm placing the file in the hmc directory, but we could also re-structure the sampling methods directory if needed.

Like mentioned in my proposal, one of the stumbling blocks in the Stan implementation was the potential fragility of 2nd and 3rd order autodiff. A first step would be to tackle this problem, or look at a different way to go about this, by maybe taking some inspiration from the MATLAB implementation.

[aseyboldt]

@bhargavvader btw, you might want to have a look at #2345. That contains a class for computing the value and gradient of a theano tensor, that merges the inputs into one array (and is typically quite a bit faster than the old version). Maybe something similar would work for the hessian (you need the full hessian, do you, or is a hessian-vector product enough?)

[bhargavvader]

Thanks @aseyboldt - I'll definitely have a look, and it should be handy. I might need the full hessian if I also do a NUTS for manifolds, for now I'm just focusing on rewriting the Hamiltonian Equations using the SoftAbs metric.
@ColCarroll , @jsalvatier - since the only real change with using manifolds is the metric, which changes the energy and velocity equations, I'm working on coding these up.

[bhargavvader]

@aseyboldt I'll be trying to use some Hessian approximations, the gradients will take forever if I don't. Just for reference, I'm trying to implement algorithm 1 and algorithm 2 on page 12 in this paper which describes a possible Hamiltonian evolution for RMHMC.

[aseyboldt]

Which approximations are you implementing? The diagonal Hessian looks useful (and I think that should be possible to compute in theano without having to go for the whole hessian and hopefully without scan as well). Footnote 3 sounds interesting, too. :-)

[bhargavvader]

Closing this PR to accommodate for @aseyboldt refactoring.

[aseyboldt]

I hope you don't have to change too much. If you need any help with that, just ping me on slack.
You can just rebase and force push to the pull request by the way.