PyMC3: Alternative Computational Engines

My Google Summer of Code 2018 project

Org: NumFOCUS

Sub-org: PyMC

Repository : https://github.com/pymc-devs/pymc4/

Mentor: Colin Carroll, Chris Fonnesbeck

Introduction

PyMC4 is a high-level probabilistic programming interface for TensorFlow. We use the capabilities of Edward2 and tensorflow_probability to make user-friendly, functional API.

Edward2 provides us with distributions and some sampling techniques. We combine these to build a user focused API.

Functional Design

Tensorflow and Edward 2 use a functional approach, where each Tensorflow program is defined as a function that takes a tensor as input and outputs a tensor. We extend this design to the PyMC4 API.

Interceptors

Edward2 wraps a tensorflow distribution as a function and makes it interceptable.

We make use of these interceptions for sampling, keeping track of variables, etc.

Collaboration with the Google tf.probability Team

The PyMC team has been coordinating with the Google tensorflow/probability team to fix existing and potential implementation problems of PyMC4 or Tensorflow Probability.

Work Done

Tested context manager based API Link

I started the project by trying to replicate the existing API of PyMC3. This design relied on context manager which kept track of random variables. I got this design to sampling stage. However we (PyMC team and I) soon realized that we are not making use of TensorFlow's functional design properly.

Functional API

The functional API of PyMC4 is an effort to extend the functional design of Tensorflow probability and Edward2 to PyMC4 and fully make use of all their capabilities.

• Took the prototype designed by Josh Safyan and adopted it to our functional design.
• Got the models to sample using HMC.
• Reproduced the famous Eight Schools Model using PyMC4 comparing it with the PyMC3 implementation. Link

Solving the Graph growth Problem

• There is problem of tensorflow graph growth while sampling or specifically while evaluating log probability. This problem has been demonstrated in this notebook and this blog post. The tensorflow team is actively helping in solving this problem.

Half Cauchy Distribution

Tensorflow Probability lacks Half Cauchy distribution which is often used in many probabilistic models. The PR for adding this to PyMC4 is in the process of being merged.

My Blog Posts

• Google Summer of Code 2018 with PyMC

• It's been a Month

• Non-Centered Eight Schools Model with PyMC4

• The Tensorflow Graph Problem

My PRs

• [MERGED] Model context manager, primitive default sampling, random variable class
• [MERGED] Get mcmc sampling to work
• [MERGED] Add credits
• [MERGED] Update Readme
• [MERGED] Merge functional
• [OPEN] A simple solution to work on the copy of the graph
• [OPEN] Add HalfCauchy distribution

Credits

As PyMC4 builds upon TensorFlow, particularly the TensorFlow Probability and Edward2 modules, its design is heavily influenced by innovations introduced in these packages. In particular, early development was partially derived from a prototype written by Josh Safyan.