How does Calibration work

[nikitaaveritchev]

Hi everyone,

i am currently trying to understand how Robyn does Calibration with lift data. In my understanding the function robyn_calibrate is used to perform calibration. However, I find the function in the code only in one place under model.R, which is used to determine the mape: calibration error. What am I misunderstanding or where does the calibration take place then?

Thanks for the help

[bsaunders23]

+1, it's been a while since I've been working with R but from the looks of it appears that the average MAPE from explicated coefficients that have priors is passed into Nevergrads optimizer. What I'm not really sure on is how the objectives are being weighted effectively (assuming uniform), going to do some more digging and see what I can find.

When setting up the optimizer (L527-L538)

  ## Start Nevergrad optimizer
  if (!hyper_fixed) {
    my_tuple <- tuple(hyper_count)
    instrumentation <- ng$p$Array(shape = my_tuple, lower = 0, upper = 1)
    optimizer <- ng$optimizers$registry[optimizer_name](instrumentation, budget = iterTotal, num_workers = cores)
    # Set multi-objective dimensions for objective functions (errors)
    if (is.null(calibration_input)) {
      optimizer$tell(ng$p$MultiobjectiveReference(), tuple(1, 1))
    } else {
      optimizer$tell(ng$p$MultiobjectiveReference(), tuple(1, 1, 1))
    }
  }

When evaluating an iteration (L855 - L865)

if (!hyper_fixed) {
            if (is.null(calibration_input)) {
              for (co in 1:iterPar) {
                optimizer$tell(nevergrad_hp[[co]], tuple(nrmse.collect[co], decomp.rssd.collect[co]))
              }
            } else {
              for (co in 1:iterPar) {
                optimizer$tell(nevergrad_hp[[co]], tuple(nrmse.collect[co], decomp.rssd.collect[co], mape.lift.collect[co]))
              }
            }
          }

[gufengzhou]

This is correct. Robyn uses the ask and tell interface from Nevergrad to interact with objective functions and hyperparameters. As you see in the code, when calibration is on, the third objective will be added. The weightage was not possible (so everything had the same weight) until recently. Now it's possible to set the weight as
optimizer$set_objective_weights(c(3,1,1)) to a 3 times weight on the first objective, for example. I'll still need to implement in Robyn though.

In general, I'd be very cautious about setting weights other than uniform unless you really know what you're doing. There's no research on how to trade off prediction vs Business goals. Also they can't be optimised (at least not in Robyn currently)

[bsaunders23]

Thanks for the quick help @gufengzhou ! I think my only thought would be whether it would be more appropriate to weight MAPE proportionally to the number of features that have priors (feels weird if I have one channel that I have some lift testing for that I'd assign as much of a cost consideration to the optimizer as prediction/business goals error).

On the subject of multi-target optimization,I still need to look into the actual optimizer being used under the hood but just going off the API notes seems like this implementation in Nevergrad is still pretty rough (maybe they just haven't updated mkdocs?). Is what it is but wondering how much of a concern that'd be w/ productionalizing.

[gufengzhou]

There's a simulated research that shows calibration with only one channel will benefit other uncalibrated channels and the whole model too.

Because causal experiment is the gold standard and the only source of causal truth (we're skeptical about any "graphical causal models" due to the ambiguity in Granger-causality), theoretically it should play a more important role than others. But as I said there's no research on weight setting yet, so I believe uniform is the fairest we can do.

Calibrated MMM better predicts true ROAS | by Gufeng Zhou - Medium https://medium.com/@gufengzhou/calibrated-mmm-better-predicts-true-roas-d5adfc8abdc4

[gufengzhou]

Reg. the doc paragraph you cited from Nevergrad, AFAIK it's ok if there're not too many (I'm talking about two digits and more) objectives, so we're ok, as I'm told by the Nevergrad team. But you could raise an issue on an update on this in Nevergrad if you want.

[bsaunders23]

Thanks for the quick response @gufengzhou , folks I talked to were 100% right on Robyn having a very active support team :). I'll take a look through, agreed on causal lift and honestly I probably need to understand Nevergrad a bit better before making suggestions. The general concern I have is that the Lift Estimate is based on a single point estimate without adjustment on confidence levels (based on the implementation it looks like confidence levels are only check to validate they're greater than 80% (see here). Could be nice to expose a hyperparameter for this or discount the importance of MAPE on a particular feature based on the supplied confidence of an experiment , I'll think about it a bit more and maybe come back to it if I can come up with an appropriate discounting/prove out the concept.