Engagement, Conversion and Retention Module

[andrewheusser]

I wanted to get feedback on the possibility of a new module focused on engagement, conversion and retention. I think something like this could fit nicely within the existing suite of pymc-marketing tools and expand the capabilities of the package to other common analyses used by product/marketing teams. I think there is a lot of value to a Bayesian approach here. In practice I've witnessed a lot of bad decisions made based rates calculated from a small amounts of data. Here's a non-exhaustive list of ideas, could start small and grow over time.

Engagement
Measure engagement with digital marketing assets. Some example metrics below:

• Websites
  • Bounce rates
  • Time on page
  • "Core" actions (e.g. download or subscribe to email)
• Social media
  • Engagement rate for different digital assets
  • Audience growth rate
  • Click through rate
  • Mentions/tags
• Email
  • Open rates
  • Click through rates
  • Unsubscribe rates

Conversion
Characterize the conversion to purchase moment as well as what that means in terms of expected revenue

• Marketing/product funnel analyses that characterize conversion rates / drop-off at each step along the way
• Modeling of conversion to purchase rate over time
• Ability to statistically compare conversion rates
• Support for modeling conversion to multiple subscription options

Retention
Characterize how many stick with the product after conversion

• Retention curves
• Retention by feature/software version
• Win-back campaigns

I think this list mostly boils down to a handful of core functions that can be repackaged/reused for each use case. Ability to:

• Model conversion rates
• Model sequential series of conversion rates (e.g. a funnel)
• Model time-series of conversion rates
• Statistically compare conversion rates for different variants (e.g. A/B tests)
• Model of retention curves
• Statistically compare retention curves for different variants (e.g. A/B tests)

This is just sort of a brain dump to get the conversation started. Please feel free to share any ideas (including if you think this is a bad idea in the first place).

[ColtAllen]

Is this intended to be more of an EDA suite? I have some ideas around covariate causality for the CLV models which seem to overlap with the bullet points in the Retention section.

[andrewheusser]

Hmm, I'm not sure I'd necessarily consider it an EDA suite. In my experience, these are all fairly common KPIs that product/marketing folks keep track of and build goals around. The value here would be in implementing these in a Bayesian framework to help emphasize uncertainty around the metrics and also leverage priors particularly in data impoverished situations. The objective of the module would be to help marketing and product analysts:

1. understand the performance of their marketing assets
2. identify leaks in their funnels
3. model conversion to purchase
4. model retention and churn

I'd love to hear your ideas on covariates for CLV models. Retention/churn models are (as you know) related to CLV, and perhaps that functionality could live in the CLV module (maybe you've already implemented what I had in mind, I will look again)

[ColtAllen]

Most of the CLV module is comprised of retention/churn models, many of which have beta-binomial assumptions. Conversion is currently unsupported, but there's an open issue for a time-to-conversion model..

I'm planning for most, if not all CLV models to support time-invariant covariates by the end of the year. Right now the only model that does so is ParetoNBDModel, but there's a PR underway for others. I'm also currently working on a subscription renewal model supporting both static and time-varying covariates.

By calculating the partial derivative for a given covariate, we can estimate the average treatment effect (ATE), and in the case of ParetoNBDModel, the conditional average treatment effect (CATE).

[cetagostini]

Hey @andrewheusser

Quick comment, what do you mean by model a conversion rate? Can you give me more input about it? Currently, I don't see a use case where rates are giving extra input or information. Based on names, looks like a model to explain middle consequences, which depends on a final output (Because its a rate of something), instead of the final consequence (output). (?)

Really like the point of retention curves, and statistically compare . . . for different variants of something. Skip the "rates" part because I need more input to make a formal opinion.

[andrewheusser]

Thanks @cetagostini - a simple model for a conversion rate is just a beta-binomial. You can string together a series of them to model conversion at each step of a funnel. I think your intuition is right on the 'middle consequences'. Imagine you download an app and there are a series of screens designed to sell you on the app prior to a paywall (i.e. an onboarding funnel). Measuring the drop off at each step in the funnel can help marketers optimize their funnel with the goal of increasing conversion to purchase. Here's a toy example of how that could be visualized for 2 variants of an onboarding funnel:

[andrewheusser]

Thinking more about this...Probably makes sense to narrow the scope here to maybe a "Conversion Funnels" module, which would be straightforward to implement and applicable across many use cases.

[wd60622]

Let's do it. Is the Binomial model a good for approach? Would there be any relationship between the parameter estimates between stages of the funnel? For instance, the 1 -> 2 influences the parameter from 2 -> 3. Independence is easy first step though

It would be straight forward to have a model on the stages and extend to ("stage", "product") or additional dimensions

[andrewheusser]

👍 Beta-Binomial for binary conversions and Dirichlet/Multinomial for multi-choice conversions (e.g. multiple subscription offerings). Could keep things simple for now and assume independence between steps. Not sure I understand your comment on the stages, can you explain?

[wd60622]

Maybe "step" is better terminalogy to refer to parts of the funnel

All I am saying is that it can be generalized pretty easily with our tooling

Think binomial model would be good start

[andrewheusser]

Following the contribution guidelines, I started preliminary work on this on a fork: https://github.com/andrewheusser/pymc-marketing/tree/conversion-funnel

[wd60622]

Can you create a PR even if it is a draft

[ColtAllen]

Some interesting points relating conversion modeling to MMM in the Recast docs.

[andrewheusser]

This definitely seems like a good direction to go (albeit more complex). Then, we could just build some helper functions to produce models/analyses/plots of the funnel. What are your thoughts @ColtAllen @wd60622 @carlosagostini

Separately, there's an opportunity to integrate CLV models here as well. At the end of the day, we want to know what channels are maximizing profits, so predicting summed CLV/LTV in lieu of sales per se (unless this is already a feature im not aware of).

[ColtAllen]

I see Recast's approach as a source of inspiration, rather than something to strictly emulate.

Conversion could be an effective bridge between MMM and CLV, because the former lacks individual customer focus and the latter can't account for new acquisitions. I might just fast-track development on #322 this month.