GPU support - examples

[springcoil]

I ran the latest master Using gpu device 0: Tesla K80 (CNMeM is disabled, cuDNN 5110) (on AWS p2.xlarge).

I used ami-e904398f (on a spot instance) and installed conda and then installed pymc3/ theano to latest master.

I ran a few experiments after installing the latest master. I'll document here what I see.

I've used the examples folder for these benchmarks. A tick confirms it runs properly

• arma_example
• baseball
• lasso_missing
• lightspeed_example
• censored_data
• gelman_schools
• GHME_2013
• custom_dists
• disaster model arbitrary deterministic
• edward_beta_bernoulli
• factor_potential
• gelman_bioassay
• LKJ Correlation
• simple test

[springcoil]

Error on lasso_missing

Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 10000000000000000159028911097599180468360808563945281389781327557747838772170381060813469985856815104.000000
         Iterations: 0
         Function evaluations: 2
         Gradient evaluations: 1
Traceback (most recent call last):
  File "lasso_missing.py", line 43, in <module>
    start = pm.find_MAP()
  File "/home/ubuntu/miniconda3/envs/bunnies/lib/python3.5/site-packages/pymc3/tuning/starting.py", line 168, in find_MAP
    specific_errors)
ValueError: Optimization error: max, logp or dlogp at max have non-finite values. Some values may be outside of distribution support. max: {'p_mother_logodds_': array(0.0, dtype=float32), 'p_disab_logodds_': array(0.0, dtype=float32), 'mother_imp_missing': array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0]), 'siblings_imp_missing': array([], dtype=int64), 'disability_imp_missing': array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), 's_log_': array(1.6094379425048828, dtype=float32), 'sib_mean_log_': array(-0.3665129244327545, dtype=float32), 'beta': array([ 0.1,  0.1,  0.1,  0.1,  0.1,  0.1,  0.1], dtype=float32)} logp: array(nan) dlogp: array([  8.98254013e+01,  -5.85000000e+01,  -9.50000000e+00,
         5.58700703e+04,   6.49721985e+02,   3.32593994e+02,
         7.25313965e+02,   1.26741997e+02,   3.48175547e+04,
         3.19725983e+02,   2.59157990e+02], dtype=float32)Check that 1) you don't have hierarchical parameters, these will lead to points with infinite density. 2) your distribution logp's are properly specified. Specific issues:
beta.logp bad: nan```

[springcoil]

Arma works it seems!

[springcoil]

Error in lightspeed example too. It seems to be very similar to the error in the baseball example. Perhaps @kyleabeauchamp or @nouiz can comment.

    return TransformedDistribution.dist(dist, self)
  File "/home/ubuntu/miniconda3/envs/bunnies/lib/python3.5/site-packages/pymc3/distributions/distribution.py", line 48, in dist
    dist.__init__(*args, **kwargs)
  File "/home/ubuntu/miniconda3/envs/bunnies/lib/python3.5/site-packages/pymc3/distributions/transforms.py", line 61, in __init__
    v = forward(FreeRV(name='v', distribution=dist))
  File "/home/ubuntu/miniconda3/envs/bunnies/lib/python3.5/site-packages/pymc3/model.py", line 800, in __init__
    self.logp_elemwiset = distribution.logp(self)
  File "/home/ubuntu/miniconda3/envs/bunnies/lib/python3.5/site-packages/pymc3/distributions/continuous.py", line 152, in logp
    value >= lower, value <= upper)
  File "/home/ubuntu/miniconda3/envs/bunnies/lib/python3.5/site-packages/theano/tensor/var.py", line 69, in __ge__
    rval = theano.tensor.basic.ge(self, other)
  File "/home/ubuntu/miniconda3/envs/bunnies/lib/python3.5/site-packages/theano/gof/op.py", line 625, in __call__
    storage_map[ins] = [self._get_test_value(ins)]
  File "/home/ubuntu/miniconda3/envs/bunnies/lib/python3.5/site-packages/theano/gof/op.py", line 562, in _get_test_value
    ret = v.type.filter(v.tag.test_value)
  File "/home/ubuntu/miniconda3/envs/bunnies/lib/python3.5/site-packages/theano/tensor/type.py", line 173, in filter
    raise TypeError(err_msg, data)
TypeError: For compute_test_value, one input test value does not have the requested type.

The error when converting the test value to that variable type:
TensorType(float32, scalar) cannot store accurately value 5331.81038143, it would be represented as 5331.810546875. If you do not mind this precision loss, you can: 1) explicitly convert your data to a numpy array of dtype float32, or 2) set "allow_input_downcast=True" when calling "function".
5331.81038143```

[junpenglao]

the lasso_missing is the start = pm.find_MAP(). Since we now have ADVI init we should use that instead.

[springcoil]

Yeah we should - wanna open a PR?

[springcoil]

For GHME I get Using gpu device 0: Tesla K80 (CNMeM is disabled, cuDNN 5110) QXcbConnection: Could not connect to display :0 - which I think is a Linux Debian error not a PyMC3 error. But I'll note it.

[springcoil]

Error on custom_dists.

Warning: Desired error not necessarily achieved due to precision loss.
         Current function value: 10000000000000000159028911097599180468360808563945281389781327557747838772170381060813469985856815104.000000
         Iterations: 0
         Function evaluations: 2
         Gradient evaluations: 1
Traceback (most recent call last):
  File "custom_dists.py", line 38, in <module>
    start = pymc3.find_MAP()
  File "/home/ubuntu/miniconda3/envs/bunnies/lib/python3.5/site-packages/pymc3/tuning/starting.py", line 168, in find_MAP
    specific_errors)
ValueError: Optimization error: max, logp or dlogp at max have non-finite values. Some values may be outside of distribution support. max: {'intercept_interval_': array(0.0, dtype=float32), 'slope': array(0.0, dtype=float32), 'sigma': array(1.0, dtype=float32)} logp: array(nan) dlogp: array([ 46097.95703125,  47503.90234375,  47796.69921875], dtype=float32)Check that 1) you don't have hierarchical parameters, these will lead to points with infinite density. 2) your distribution logp's are properly specified. Specific issues:
intercept_interval_.logp bad: nan``` 



Cc @junpenglao 

[springcoil]

Disaster model fails too

    rate = rateFunc(switchpoint, early_mean, late_mean)
  File "/home/ubuntu/miniconda3/envs/bunnies/lib/python3.5/site-packages/theano/gof/op.py", line 615, in __call__
    node = self.make_node(*inputs, **kwargs)
  File "/home/ubuntu/miniconda3/envs/bunnies/lib/python3.5/site-packages/theano/gof/op.py", line 966, in make_node
    (str(self.itypes), str([inp.type for inp in inputs])))
TypeError: We expected inputs of types '[TensorType(int64, scalar), TensorType(float64, scalar), TensorType(float64, scalar)]' but got types '[TensorType(int64, scalar), TensorType(float32, scalar), TensorType(float32, scalar)]'```

[junpenglao]

I know @ferrine is working on changing the ADVI init to the new ADVI API - maybe we should wait after that?