q_0 distributions using toys

[SimonBerlendis]

Dear pyhf experts,

I am trying to do a short-and-simple statistic test using pyhf (first time for me!). But I end up with a weird result. Can someone help me to understand what I did wrong ?

I want to do a q_0 statistic test using a simple model with one single signal region and no systematic uncertainties and I would like to calculate the signal significance using toys.

I created a jupyter notebook which you can find a version here: https://sberlend.web.cern.ch/Test.html

To summarize:

I generated a simple model like this:

model = pyhf.simplemodels.hepdata_like(signal_data=[30.0], bkg_data=[100], bkg_uncerts=[0])
observations = [130]
data = pyhf.tensorlib.astensor(observations + model.config.auxdata)

I created the test statistics by doing the following :

toy_calc = pyhf.infer.calculators.ToyCalculator(data, model, test_stat='q0', ntoys=10000)

And then I generated the q_0 distributions using the following method:

sb_dist, b_dist = toy_calc.distributions(poi_test=0)

I end up with the following distributions:

Which seems wrong, since I would expect a distribution peaked at zero and rapidly falling for high q_0 values. Did I did something wrong ?

Cheers, Simon.

[matthewfeickert]

What version of pyhf are you using?

Without having fully looked at this at all yet, I'll note first that bkg_uncerts=[0] is not correct. In your notebook you say that an uncertainty of 0 means a statistical uncertainty only, but that's not right — if you want a statistical uncertainty only then it should be (for pyhf v0.6.1)

model = pyhf.simplemodels.hepdata_like(signal_data=[30.0], bkg_data=[100], bkg_uncerts=[10])

Check out the "Running Monte Carlo simulations (toys)" example notebook "Counting Experiment" section for an example of this being done.

Could you also make your notebook and environment file available as a GitHub Gist?

[SimonBerlendis]

Hi Matthew,

I use v0.6.1.

I recently used the method described in 'Running Toys by Hand' in your link and it worked fine. See the q_0 distribution below.

I wonder if the ToyCalculator does the correct job for q_0 tests...

Related to your comment on the uncertainty, I thought that in practice, this would be defined as a Gaussian term in the likelihood function. If we assume a null uncertainty on the background, the likelihood function would be defined as 𝐿(𝜇)=Pois(𝑁|𝜇𝑆+𝐵), which takes into account the poisson's statistical uncertainty. Therefore, my statistical test should still be stat-only, don't you think ?

[SimonBerlendis]

I tried to create the github gist with the new version of the notebook. That's the first time I use this functionality, so tell me if it didn't work :
https://gist.github.com/SimonBerlendis/42974be4057fd7681f68c54929cea14b