feat: custom modifiers

[lukasheinrich]

Description

This is a first example on how to e.g. have a "gaussian" bump on top of a simple model
this gives a first look into how to solve #850

cc @alexander-held

#%%
import pyhf
import matplotlib.pyplot as plt
import numpy as np
# %%
class custom_modifier(object):
    def apply(self,pars):
        #shape is  (n_modifiers, n-global_samples, n_alphas, n_global_bin)
        base = np.zeros((1,len(m.config.samples),1,sum(m.config.channel_nbins.values())))

        import scipy.stats
        bins = np.linspace(-5,5,11)
        yields = 20*(scipy.stats.norm.cdf(bins[1:]) - scipy.stats.norm.cdf(bins[:-1]))

        base[0,self.config.samples.index('signal'),0,:] = yields
        return base



m = pyhf.Model(
{'channels': [{'name': 'singlechannel',
   'samples': [{'name': 'signal',
     'data': [0]*10,
     'modifiers': [{'name': 'mu', 'type': 'normfactor', 'data': None}]},
    {'name': 'background',
     'data': [10]*10,
     'modifiers': [{'name': 'uncorr_bkguncrt',
       'type': 'shapesys',
       'data': [5]*10}]}]}]},
       custom_modifiers=[custom_modifier()]
)

bp = pyhf.tensorlib.astensor(m.config.suggested_init())
bp[m.config.poi_index] = 0.2
plt.bar(range(10),m.expected_actualdata(bp))
# %%

# %%

[alexander-held]

A big question for solving #850 is what kind of language to use to specify custom expressions. I think for many use cases the expressions are simple enough that the language does not matter much. And it might be possible to add the language to the workspace in such a way that in the future new languages could be added?

[lukasheinrich]

yeah I think we need to solve 2 issues:

• just technically allow the specification of new modifiers at the python level
• how to codify these into JSON this is in the direction of HiFav2

a big question is also in what order these modifications happen, i.e. here we have the custom modifier an additive modifier where the total rate in pyhf is computed as

lambda = (nominal + additive.....) * multiplicative

but you could also imagine expressions that expect to operate on yields onec all other modifiers have been applied

[alexander-held]

I think this is technically still in the scope of normal HistFactory https://root.cern/doc/v620/classRooStats_1_1HistFactory_1_1Measurement.html#ade80531b4588d22ad5e7339ce74e7027. The version implemented there just acts as a normfactor, so this question of order is avoided. Once you allow more custom variations the problem becomes more complicated indeed.

[lukasheinrich]

this now supports declaring additional parameters in the modifiers.. here e.g. a mean of the gaussian

class custom_modifier(object):
    def __init__(self,name = 'name'):
        self.required_parsets = {
            'mean': {
                'paramset_type': pyhf.parameters.paramsets.unconstrained,
                'n_parameters': 1,
                'modifier': 'normfactor',
                'is_constrained': False,
                'is_shared': True,
                'inits': (0.0,),
                'bounds': ((-5, 5),),
                'fixed': False
        }}
    def apply(self,pars):
        base = np.zeros((1,len(m.config.samples),1,sum(m.config.channel_nbins.values())))

        bins = np.linspace(-5,5,11)
        mean = pars[self.config.par_slice('mean')][0]
        yields = 20*(scipy.stats.norm(loc = mean).cdf(bins[1:]) - scipy.stats.norm(loc = mean).cdf(bins[:-1]))
        base[0,self.config.samples.index('signal'),0,:] = yields
        return base

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[balunas]

It looks like it's been quite some time since any progress was made here. It's been bumped back for a few releases, is there some technical limitation, or has it just not been prioritized? There are some analysis use cases this could be really helpful for, and we've been having to use some really inefficient work-arounds instead. Is there a way we could help?

[IzaV10]

Hi,

I tried running the simple example with the "gaussian" bump on top of a simple mode. The branch was installed as pip install git+https://github.com/scikit-hep/pyhf.git@custom_modifiers but when running the code abovel (from the very first entry of the conversation) I get an error.

File "/<PATH TO VIRTUAL ENVIRONMENT>/pyhf-env/lib/python3.7/site-packages/pyhf/pdf.py", line 497, in __init__
    self.config = _ModelConfig(self.spec, **config_kwargs)
  File "/<PATH TO VIRTUAL ENVIRONMENT>/pyhf-env/lib/python3.7/site-packages/pyhf/pdf.py", line 171, in __init__
    f"Unsupported options were passed in: {list(config_kwargs.keys())}."
pyhf.exceptions.Unsupported: Unsupported options were passed in: ['custom_modifiers'].

I suspect that the way the class custom_modifier() is defined needs to be changed. Is there any documentation available on how to use this option?

Thank you for your help and time.

Cheers,
Iza

[lukasheinrich]

tests pass (2 tiny floating point changes... ) will clean up tomorrow

[lukasheinrich]

I don't really understand the CI failures - passes on my machine - any clues @matthewfeickert ?

[lukasheinrich]

closed in favor of #1625