Attribute error when observed has multiple dimensions: MultiObservedRV

[sidravi1]

Here's a Weibull survival model in pymc3:

def weibull_lccdf(alpha, beta, x):
    return -(x / beta)**alpha

class Weibull_like(pm.Continuous):
    def __init__(self, alpha, lambd, *args, **kwargs):
       
        self.alpha = tt.as_tensor_variable(alpha)
        self.lambd = tt.as_tensor_variable(lambd)
        
        super().__init__(*args, **kwargs)

    def logp(self, failure, value):
        
        alpha = self.alpha
        beta = 1/tt.exp(self.lambd)

        no_censoring = failure.astype('bool')
        beta_nocens = beta[no_censoring]
        beta_censor = beta[~no_censoring]
        
        uncensored = pm.Weibull.dist(alpha, beta_nocens).logp(value[no_censoring]).sum()
        censored = weibull_lccdf(alpha, beta_censor, value[~no_censoring]).sum()
        
        return uncensored + censored

and the setting up the model

with pm.Model() as weibull:
    beta0 = pm.Normal('β_0', 0, 100)
    beta1 = pm.Normal('β_1', 0, 100)
    alpha = pm.Gamma('α', 1.0, 1/0.001)
    
    lam_ = pm.Deterministic('λ', beta0 + beta1*(treatment - treatment.mean()))
    
    like = Weibull_like('survival', alpha=alpha, lambd=lam_, observed={'failure':failure, 'value':t})
    
    trace_weibull = pm.sample(tune=1000, draws=3000, chains=2)

Running any of plots gives an Attribute error:

av.plot_trace(trace_EVM)

/anaconda3/lib/python3.6/site-packages/arviz/data/io_pymc3.py in <dictcomp>(.0)
    107         model = self.trace._straces[0].model  # pylint: disable=protected-access
    108 
--> 109         observations = {obs.name: obs.observations for obs in model.observed_RVs}
    110         if self.dims is None:
    111             dims = {}

AttributeError: 'MultiObservedRV' object has no attribute 'observations'

Poking around a bit, it looks like MultiObservedRV, doesn't create an observations parameter but instead saves in self.data:

https://github.com/pymc-devs/pymc3/blob/master/pymc3/model.py#L1369

Suggestion on how to fix this? Is it as simple as just patching the PyMC3Converter class where it build the observations dict?

https://github.com/arviz-devs/arviz/blob/master/arviz/data/io_pymc3.py#L110

I'm assuming just doing this will break some of the plots?

[ahartikainen]

Will it create correct InferenceData object?

[ColCarroll]

Darn, I'd never even seen MultiObservedRV.

I think patching there will work, and will probably break a few plots that assume there is only 1 observed variable. Tests will continue to pass, since nothing checks for this.

Maybe a solution is to add a survival model (like this, but with your likelihood) to the model zoo, and include plot(s) from it in the documentation. Is the model you're working with public, or could it be?

[sidravi1]

Ok cool. I’ll make the changes and do a PR. And maybe add a test case or two. Happy to add it to the model zoo. Right now I’m running it on a publicly available dataset as a POC so happy to share.

…

On Wed, Oct 31, 2018 at 11:46 AM Colin ***@***.***> wrote: Darn, I'd never even seen MultiObservedRV. I think patching there will work, and will probably break a few plots that assume there is only 1 observed variable. Tests will continue to pass, since nothing checks for this. Maybe a solution is to add a survival model (like this <https://docs.pymc.io/notebooks/survival_analysis.html>, but with your likelihood) to the model zoo, and include plot(s) from it in the documentation. Is the model you're working with public, or could it be? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#384 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ANeLhZjJvsZ7LuVU4rRx8HipzanuamdUks5uqcXhgaJpZM4YEUUJ> .

[FlorianWilhelm]

I just ran into the same problem.

[mfansler]

Hit the same issue when using pymc3.DensityDist for making a weighted model. FWIW, here is the reproducible example I made for this.

import numpy as np
import pymc3 as pm
import arviz as az

# fake regression data, with weights (W)
np.random.seed(2019)
N = 100
X = np.random.uniform(size=N)
W = 1 + np.random.poisson(size=N)
a, b = 5, 17
Y = a + np.random.normal(b*X)

with pm.Model():
    a = pm.Normal('a', 0, 10)
    b = pm.Normal('b', 0, 10)
    
    mu = a + b*X
    sigma = pm.HalfNormal('sigma', 1)
    
    def weighted_normal(y, w):
        return w*pm.Normal.dist(mu=mu, sd=sigma).logp(y)
    
    y_logp = pm.DensityDist('y_logp', weighted_normal, observed={'y': Y, 'w': W})
    
    trace = pm.sample(2000, tune=2000, random_seed=2019)

traz = az.from_pymc3(trace)

which gives the error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-50-3aba64ad8aca> in <module>
----> 1 traz = az.from_pymc3(trace)

~/miniconda3/envs/pymc36/lib/python3.7/site-packages/arviz/data/io_pymc3.py in from_pymc3(trace, prior, posterior_predictive, coords, dims)
    150         posterior_predictive=posterior_predictive,
    151         coords=coords,
--> 152         dims=dims,
    153     ).to_inference_data()

~/miniconda3/envs/pymc36/lib/python3.7/site-packages/arviz/data/io_pymc3.py in to_inference_data(self)
    138                 "posterior_predictive": self.posterior_predictive_to_xarray(),
    139                 "prior": self.prior_to_xarray(),
--> 140                 "observed_data": self.observed_data_to_xarray(),
    141             }
    142         )

~/miniconda3/envs/pymc36/lib/python3.7/site-packages/arviz/data/base.py in wrapped(cls, *args, **kwargs)
     23                 if getattr(cls, prop) is None:
     24                     return None
---> 25             return func(cls, *args, **kwargs)
     26 
     27         return wrapped

~/miniconda3/envs/pymc36/lib/python3.7/site-packages/arviz/data/io_pymc3.py in observed_data_to_xarray(self)
    108         model = self.trace._straces[0].model  # pylint: disable=protected-access
    109 
--> 110         observations = {obs.name: obs.observations for obs in model.observed_RVs}
    111         if self.dims is None:
    112             dims = {}

~/miniconda3/envs/pymc36/lib/python3.7/site-packages/arviz/data/io_pymc3.py in <dictcomp>(.0)
    108         model = self.trace._straces[0].model  # pylint: disable=protected-access
    109 
--> 110         observations = {obs.name: obs.observations for obs in model.observed_RVs}
    111         if self.dims is None:
    112             dims = {}

AttributeError: 'MultiObservedRV' object has no attribute 'observations'

---

Environment Info

PyMC3 3.6
ArviZ 0.3.3

[OriolAbril]

@sidravi1 @mfansler @ColCarroll I was checking the state of this issue, and found out that latest release does not raise an error as it used to happen. There is no observed data though. I tried to modify io_pymc3 to retrieve this kind of data but I am not sure this is the right approach, I am not eve sure it would be helpful to have this kind of data stored. It would be great if somebody could try the code or review the PR.