Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

extend model arg usage in io_pymc3 to fix plot_ppc with prior #1045

Merged
merged 10 commits into from
Feb 11, 2020

Conversation

OriolAbril
Copy link
Member

@OriolAbril OriolAbril commented Feb 5, 2020

Description

Try to improve handling of prior groups and observed data in io_pymc3. Related to #1002. It introduces an important change in behaviour, it proposes to exclude the observed_data group whenever predictions are present. @corriebar @rpgoldman

I also extended tests on io_pymc and improved the helper function check_multiple_attrs.

And finally, how do you feel about deprecating the use of from_pymc3 without model nor trace (or even without model). I was thinking on adding a warning. For now not using a model nor trace will raise a pending deprecation warning as a start. @canyon289 @aloctavodia @ColCarroll

Related to #939.

Checklist

  • Does the PR follow official
    PR format?
  • Is the new feature properly documented with an example?
  • Does the PR include new or updated tests to cover the new feature (using pytest fixture pattern)?
  • Is the code style correct (follows pylint and black guidelines)?
  • Is the change listed in changelog?

@OriolAbril OriolAbril changed the title Io pymc model extend model arg usage in io_pymc3 to fix plot_ppc with prior Feb 5, 2020
@@ -121,8 +121,7 @@ def arbitrary_element(dct: Dict[Any, np.ndarray]) -> np.ndarray:
def find_observations(self) -> Optional[Dict[str, Var]]:
"""If there are observations available, return them as a dictionary."""
has_observations = False
if self.trace is not None:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really like leaving out the check that tells the caller why they are not getting observations out of their trace when they expect it. Perhaps replace my error with a UserWarning?
It seems like Arviz tries to get whatever information it can out of a trace, and ignores whatever it can't figure out. This makes it relatively robust, but means that the user doesn't know that they could get more information into the InferenceData by providing a model.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I completely agree, I was just waiting in order to replace it with an informative warning or a deprecation warning.

@rpgoldman
Copy link
Contributor

rpgoldman commented Feb 5, 2020

Per my comment in the code and your question in the MR description -- I agree it would be a good idea to require supply of the model. I think this is particularly important because the user could get an inappropriate model out of the trace.
This may seem far-fetched, but for PyMC3 one actually has to move a trace to a new model in some cases in order to do out of sample predictions.

@OriolAbril OriolAbril changed the title extend model arg usage in io_pymc3 to fix plot_ppc with prior [WIP] extend model arg usage in io_pymc3 to fix plot_ppc with prior Feb 6, 2020
@OriolAbril
Copy link
Member Author

Only the model argument warning-deprecation is left to do, after that we can merge

@OriolAbril
Copy link
Member Author

I decided to start with the deprecation on no model nor trace (it will be triggered in cases such as only prior or only posterior predictive/predictions). We can make it more strict next release by requiring a model even when the trace is present and eventually remove the _straces trick. In my opinion, PyMC3 users should not mind having to call from_pymc3 within a model context or passing the model as kwarg, it already is the workflow for sample, sample_prior_predictive...

I think it is ready to merge

@rpgoldman
Copy link
Contributor

I agree. If the tests pass again, we should merge. Looks very good!

@OriolAbril OriolAbril changed the title [WIP] extend model arg usage in io_pymc3 to fix plot_ppc with prior extend model arg usage in io_pymc3 to fix plot_ppc with prior Feb 10, 2020
@OriolAbril OriolAbril merged commit d33192e into master Feb 11, 2020
@OriolAbril OriolAbril deleted the io_pymc_model branch February 11, 2020 22:34
@OriolAbril
Copy link
Member Author

This allows to make prior predictive checks right away with PyMC3!

with pm.Model():
    # define model
    prior = pm.sample_prior_predictive()
    idata = az.from_pymc3(prior) # has to be either from a model context or
                                 # passing model=model to from_pymc3

az.plot_ppc(idata, group="prior")

I am not really sure about the way to advertise this though, is there a PyMC3 tutorial on prior predictive sampling? @canyon289 @aloctavodia

@AlexAndorra
Copy link
Contributor

This is great -- thanks guys!!
To my knowledge, there is no tutorial dedicated to prior pred checks on PyMC's website. I found:

  • The putting notebook, where Colin does some prior checks -- but maybe too complicated a use-case for an introduction.

  • I updated the radon NB a few weeks ago, and among other things added prior checks. The new version is not up yet (but it's on master if you wanna check it out), but this could be a good avenue to advertise this new use.

  • One NB dedicated to posterior predictive checks. We could extend (and update) it -- this would make a useful educational ressource to point to.

In general, I noticed on the Discourse that people find it hard to understand the concept of prior/posterior pred checks -- or just don't know about them. So I think an intro tutorial would be really useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants