rfm_summary: frequency and monetary_value sum or avg

[BeiYaaa]

Hi there,

I noticed that the sample dataset used in the quick start has more than one transaction for a given day of a customer history. the frequency calculated by the rfm_summary function didn't consider that. I wonder what you would do for datasets that don't have more granularity for date time.

Besides, the monetary value is supposed to be the average order value. But given the multiple transactions for a certain day, would you prefer to average the transaction values and use frequency +1 for that day, or would you lean over to sum the values?

I'm interested in if you have thoughts regarding this part. And what would be the best practice?

Thank you!

[ColtAllen]

Hey @BeiYaaa,

I believe my response to #268 (comment) answers your first question. The short answer is time between transactions is a key component of this model, and we can't go more granular than the raw data. If the raw data were hourly timestamps and rfm_summary(time_unit='D', *args), it would be counted as multiple transactions.

Your second question is a good one, and something I worked on last week in fact for an upcoming PR - an rfm_train_test_split function, because the equivalent function in the legacy lifetimes library has a bug and doesn't aggregate properly.

All transactions occurring in the same time period will have their monetary values summed together for that time period. The averaging occurs across all purchase time periods except for the first purchase. If the customer only made one purchase, then frequency=0 because the CLV model is predicting the number of repeat purchases.

Monetary values for one-time customers are also zeroed out because the GammaGammaModel can only be fit to repeat purchases. The predictions it makes for one-time customers will be the population average of the fitted Gamma distribution (which is not a simple (sum/n)).

That being said, there is an include_first_transactions parameter in rfm_summary, but it is only intended to be used for RFM Segmentation Analysis, a PR I want to start work on in March.

[BeiYaaa]

Thank you @ColtAllen it's good to know the new functions are coming.

For the rfm_train_test_split function, will it be similar to the old calibration_and_holdout_data function? Is this the equivalent you mentioned?

Currently what I'm doing is to split the entire transaction history and compare the future average values vs. the expected monetary value. Let's say I have data from 2023-01-01 to 2024-01-01. I would split the training from 2023-01-01 to 2023-06-01 and then predict the monetary value for each customer. Comparing the average transaction values of each customer that I already predicted during the testing period 2023-01-01 to 2024-01-01, I can get a sense of how accurate each customer is predicted. However, I didn't find a good way to give a one-number metric to indicate accuracy. I checked the tutorial of the old lifetimes package, it seems only the compare chart of plot_calibration_purchases_vs_holdout_purchases was used. I used RMSE for the train and test. While waiting for you to publish the new function, I wonder if you feel this would be an alternative way.

Another question, sorry for diverging from the original questions, but how would you recommend tuning the model? It seems that no matter how I set up the model, the fitted results are no way close to ideal. I wonder what parameters you use to improve the model performance.

from pymc_marketing import clv
import arviz as az
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

selected_train_df_gg = selected_train_df[selected_train_df['monetary_value']>0].copy()
selected_train_df_gg['mean_transaction_value'] = selected_train_df_gg['monetary_value']

gg = clv.GammaGammaModel(
    data = selected_train_df_gg,
    model_config={
                    "p_prior": {'dist': 'HalfNormal', 'kwargs': {"sigma": 10}},
                    "q_prior": {'dist': 'HalfStudentT', 'kwargs': {"nu": 4, "sigma": 10}},
                    "v_prior": {'dist': 'HalfCauchy', 'kwargs': {"beta":1}}
                    }
    )
gg.build_model()
gg.fit(fit_method='map')
gg.fit_summary()

expected_spend = gg.expected_customer_spend(
    customer_id=selected_train_df['customer_id'],
    frequency=selected_train_df['frequency'],
    mean_transaction_value=selected_train_df['monetary_value']
)
selected_test_df_exp_rev = pd.merge(rfm_data_test_all, pd.DataFrame({'customer_id': selected_train_df['customer_id'], 'expected_spend': expected_spend.mean(("chain", "draw")).values}),
        how='right',
        right_on='customer_id',
        left_on='customer_id')
selected_test_df_exp_rev.plot.scatter(x="monetary_value", y="expected_spend")

[ColtAllen]

For the rfm_train_test_split function, will it be similar to the old calibration_and_holdout_data function? Is this the equivalent you mentioned?

Yes.

Another question, sorry for diverging from the original questions, but how would you recommend tuning the model?

First off, did you check the correlation between purchase frequency and monetary value? It should be less than 0.3.

You can try different prior configurations, but if your dataset is larger than 30k customers, the differences will be minimal.

It's important to note the GammaGammaModel is essentially a weighted average of the underlying Gamma distributions - if a customer has only made one purchase, their expected spend is the population average, but if a customer has made many purchases, their expected spend will be biased towards their own historical average.

A recent idea I had was to fit a transaction model to the full dataset, then segment the dataset and fit a GammaGammaModel to each segment. I'll start on this PR after the rfm_train_test_split function is done.

[BeiYaaa]

First off, did you check the correlation between purchase frequency and monetary value? It should be less than 0.3.

Yes, I did. Please see below for the correlation score. The cluster column is just an extra that used when I was testing clustering the customers. Although what would be the impact if the correlation is higher than .3? I assumed that if there is a correlation between frequency and monetary value, for customer with higher monetary value or frequency, the predicted weighted spend would lean towards the higher.

You can try different prior configurations, but if your dataset is larger than 30k customers, the differences will be minimal.

Yeah, the dataset I have has 290K customers. I think the extremes made the low performance of the GammaGammaModel. What I observed from the real datasets I have is that if all customers are treated the same way, it's hard to fit a good GammaGammaModel and expected_customer_lifetime_value model, but not necessary for BetaGeoModel. Why is that? I have tried to round up the real monetary value to tiers, like $10, $20 instead of $10.5, $19.9. But it doesn't help.

A recent idea I had was to fit a transaction model to the full dataset, then #523 and fit a GammaGammaModel to each segment.

May I ask is there any particular reason you want to use one transaction model to the full dataset instead of for each segment? I tried HDBSCAN for clustering, however, the large number of clusters made it hard to iterate the GammaGammaModel for each of them. What's your experience of setting up the clusters? Would you prefer to lower the number of clusters or utilize a higher CPU/memory machine?

Thank you again @ColtAllen for sharing!

[ColtAllen]

Although what would be the impact if the correlation is higher than .3?

I haven't seen this much in practice, but if the model still converges I'd imagine predicted spend would increase with purchase frequency, regardless of the history of the customer.

What I observed from the real datasets I have is that if all customers are treated the same way, it's hard to fit a good GammaGammaModel and expected_customer_lifetime_value model, but not necessary for BetaGeoModel. Why is that?

May I ask is there any particular reason you want to use one transaction model to the full dataset instead of for each segment?

BetaGeoModel and ParetoNBDModel both have strong statistical assumptions about the customer population. If you start segmenting too soon, you risk violating these assumptions. It's best to start with the whole dataset, then filter as-needed to improve performance. You can use the code here to visually evaluate the model.

GammaGammaModel however, seems better suited for a homogenous set of customers, as you've discovered. expected_customer_lifetime_value is simply a utility for this model.

What's your experience of setting up the clusters? Would you prefer to lower the number of clusters or utilize a higher CPU/memory machine?

RFM Segmentation is my preferred approach, and yet another upcoming PR. In the meantime I'd suggest no more than 7 clusters. Try another algorithm too because HDBSCAN will throw out customers that don't quite fit anywhere, which I don't think you want.