Plotting normalization uncertainties, and post-fit impact

[Tomoya-Iizawa]

I think cabinetry.visualize.templates produces plots for shape uncertainties and not for normalization uncertainties.
Is there any way to produce plots to show the impact of the normalization uncertainties?

And this function shows the pre-fit distribution of the variation samples, right?
If so, is there any way to show the post-fit?

Best,
Tomoya

[alexander-held]

The visualize.templates function runs over existing histograms. Pure normalization variations need no histograms, which has lead to this not being supported thus far. The plots for normalization variations contain less information (just nominal scaled up/down by the value set in the config for the systematic) but I agree they can also be useful to have. I opened #474 to add support for this type of variation. Could you please give it a try and let me know if it works as intended?

This functionality is indeed all pre-fit. To do the same post-fit we probably would want to change approach and start all the visualizations from a workspace + associated pyhf model. Given a fit results object, it is straightforward to get all the relevant counts per bin (this is what model_utils.yield_stdev already does at the beginning for the yield uncertainty calculation). The up_variations and down_variations arrays in there together with the nominal post-fit prediction contain all relevant data and would just need to be plugged back into a visualization function (together with some logic to skip variations that are zero). I opened #473 to track this.

One downside of this workspace-based visualization is that some information is no longer available, namely the statistical uncertainties for variations (relevant e.g. for two-point systematic variations). Additionally, the effect of optional template histogram post-processing can also no longer be shown, as only the final templates that enter the workspace are available.

[alexander-held]

If you clone the repository, that script will run a very simple setup of a full workflow, including template visualization. If you have the correct version of cabinetry the output should include the following:

INFO - cabinetry.visualize.utils - saving figure as figures/templates/Signal_region_Signal_Luminosity.pdf
INFO - cabinetry.visualize.utils - saving figure as figures/templates/Signal_region_Background_Luminosity.pdf

which indicates that you have successfully picked up the changes for visualization normalization uncertainties. If that does not show up, I assume somehow you are picking up the wrong version of cabinetry.

[Tomoya-Iizawa]

thanks, hmm, I am using pip install as shown above
pip install git+https://github.com/scikit-hep/cabinetry.git@feat/normsys-templates
I guess it works since if I try with random name as normsys-template, it fails.
but normalization uncertainties are not shown..
in he execution file, I am doing

cabinetry.templates.collect(config)
cabinetry.templates.postprocess(config)
                    cabinetry.visualize.templates(
                        config,
                        figure_folder=figurespath,
			close_figure=True)

will this work? or cabinetry.templates.build(config, method="uproot") should be used instead of "collect"?
If I use "build" it fails with error as following, does this mean I cannot use correct branch?

  File "/afs/cern.ch/work/t/tomoya/diTauAnalysis/fitting/run_cabinetry.py", line 1072, in <module>
    run_cabinetry(args["inputs"],
  File "/afs/cern.ch/work/t/tomoya/diTauAnalysis/fitting/run_cabinetry.py", line 119, in run_cabinetry
    cabinetry.templates.build(config, method="uproot")
  File "/afs/cern.ch/work/t/tomoya/diTauAnalysis/setupMiniforge/miniforge3/lib/python3.9/site-packages/cabinetry/templates/__init__.py", line 47, in build
    route.apply_to_all_templates(
  File "/afs/cern.ch/work/t/tomoya/diTauAnalysis/setupMiniforge/miniforge3/lib/python3.9/site-packages/cabinetry/route.py", line 342, in apply_to_all_templates
    default_func(region, sample, systematic, template)
  File "/afs/cern.ch/work/t/tomoya/diTauAnalysis/setupMiniforge/miniforge3/lib/python3.9/site-packages/cabinetry/templates/builder.py", line 301, in _create_histogram
    pos_in_file = _position_in_file(sample, systematic, template)
  File "/afs/cern.ch/work/t/tomoya/diTauAnalysis/setupMiniforge/miniforge3/lib/python3.9/site-packages/cabinetry/templates/builder.py", line 228, in _position_in_file
    position = sample["Tree"]

[alexander-held]

The build function is for ntuple inputs, collect is for histogram inputs. If you are using histogram inputs, build will not work. I cannot think of a reason right now why visualize.templates would work for ntuple inputs and not for histograms. Could you please clone the cabinetry repository and see if the example.py script produces the figures for the Luminosity uncertainty? That should work if you have the right version from the branch.

[Tomoya-Iizawa]

ok, no, no, it fails with "build" and works for "collect", and I am using histogram inputs, so it seems reasonable.
ok, I will try from clone and checkout.

[Tomoya-Iizawa]

hm, in actually, I am usually running cabinetry on lxplus which requires several setups such as many pip installs that I am not sure which are really necessary.
could you tell me how to setup cabinetry on lxplus from scratch, or could you merge this function into feat/custom-modifier-models?