Lift test calibration - Data inputs

[raz1470]

I'm trying to get my head round how we would transform realistic geo-lift test results ready for mmm.add_lift_test_measurements. It looks great but I fear it will get misused without some more docs on it.

Example geo-lift test:

• Test length 8 weeks
• Marketing blackout for Facebook in 20% of regions (test)
• Marketing spend for Facebook relatively stable pre and post test for the remaining 80% of regions (control)
• Synthetic control used to estimate the drop in sales in the test regions (where the blackout was)

Transforming the data:

• Lets assume our MMM model includes weekly observations - First of all we need to divide the spend by the test length (8 weeks in our case) to get it to a weekly estimate
• Lets assume the 20% of regions where representative of country performance - Next we need to scale the spend up to country level, so in our case we multiply by 5 (as we used 1/5 of the country)
• We also need to do both of these things with the sales lift (or sales drop in our case) to get it weekly and at a country level

The things I want to clarify are:

1. Should we also be applying the adstock and saturation transformation before we pass x and delta_x (pre-test spend and changes made to spend) to mmm.add_lift_test_measurements?
2. Do we need to apply the marketing spend and target transformer (scaling by the max absolute value from the training data)?
3. Does it make a difference whether I (1) say this is a spend increase with associated sales increase (e.g. £0 -> £10k) (2) say this is a spend decrease (e.g. £10k -> £0) with associated sales decrease?
4. In terms of informing the sigma, are there any suggestions how to do this? I assume you are thinking people will use a bayesian approach to synthetic controls and have a measure of uncertainty? In my head i'm struggling to tie this in with a gamma distribution...

[wd60622]

For the items to clarify:

1 & 2 pass them as is. The transformations and scaling will be applied in the method based on build_model method.
3. I think that they would be treated the same. Do you think otherwise?
4. This sigma is in the original scale of the target as well. If you don't have one, then it might be generate from a percent of the delta_y based on what you find reasonable. The gamma distribution just captures that an increase in spend will have an increase in return (because the saturation transformation is monotonic)