Better default arguments for MCMC, SNLE, SNPE.

[michaeldeistler]

• especially num_chains
• vectorized sampler is not default
• Add a tutorial

[janfb]

Note: slice_np_vectorized with a small thin, e.g., <5, results in 0 and inf samples ❗ even for Gaussian examples and only for num_chains>1.

Update: thinning does not seem to be the problem. if thinning is small it just increases the likelihood of hitting an inf or NaN value.
The main reason seems to be the number of warmup steps. If it is too small, chains have not converged and samples diverge (why?)

[gmoss13]

Additional context: The default parameters are defined here. Following discussion in #924 - a good suggestion for the default parameters for MCMCPosterior might be:
method="slice_np_vectorized", num_chains=20, warmup_steps=50, thin=5
It could be useful to add a warning that sampling could be slow if the user sets the method to be non-vectorised, and num_workers is small.

[gmoss13]

Once #908 is addressed there might be even better default args

[janfb]

@famura can you please add the new info about best practice regarding thinning that you mentioned earlier today?
That'd be great! 🙏

[famura]

@famura can you please add the new info about best practice regarding thinning that you mentioned earlier today?
That'd be great! 🙏

Sure, where do you want me to add it? in the top level class?

[janfb]

@famura can you please add the new info about best practice regarding thinning that you mentioned earlier today?
That'd be great! 🙏

Sure, where do you want me to add it? in the top level class?

No, just post a link to the resource giving more details. Then we can use it when we decide for better default args later

[famura]

So, @felixp8 and I came across this post. Moreover, @michaeldeistler and @sethaxen discussed with the somewhat confident result that we can set thinning to 1 by default. However, since this will most likely lead to results that differ from the benchmark, we came up with the solution that we set thin to -1 by default, catch that and set it then to 1 and raise a warning that the default has changed.

Is that the kind of information you were looking for?

[famura]

On a different note @sethaxen said that num_chains=4 would be a good default

[famura]

Linked to PR #987 and PR #1053 which has the changes

[janfb]

TODO:

• switch to slice_np_vectorized as default arg (in posterior classes and build_posterior methods):
• switch to num_chains=10 as default
• set MAF for SNLE and NSF for SNPE.

A bit unrelated but I also suggest switching training defaults to

• training_batch_size=100 or even 200