Adding data keyword to validphys

doc/sphinx/source/vp/dataspecification.md

@@ -0,0 +1,103 @@
+# Data Specification
+
+## Specifying a dataset
+
+In a validphys runcard a single data is specified using a `dataset_input`. This
+is a dictionary which minimally specifies the name of the dataset, but can also
+control behaviour such as contributions to the covariance matrix for the dataset
+and NNLO cfactors.
+
+here is an example dataset input
+
+```yaml
+dataset_input:
+    dataset: CMSZDIFF12
+    cfac: [QCD,NRM]
+    sys: 10
+```
+
+This particular example is for `CMSZDIFF12` dataset, the user has specified to
+use some cfactors `cfac` and `sys: 10` which correponds to an additonal
+contribution to the covariance matrix accounting for statistical fluctuations in
+the cfactors. These settings correspond to NNLO predictions amd so presumably
+elsewhere in the runcard the user would have specified a NNLO theory - such as
+theory 53.
+
+Clearly there is a big margin for error when manually entering `dataset_input`
+and so there is a [project](https://github.com/NNPDF/nnpdf/issues/226) which aims to have a stable way of filling many of
+these settings with correct default values.
+
+## Specifying Multiple datasets
+
+Multiple datasets are specified using `dataset_inputs` key: a list where
+each element of the list is a valid `dataset_input`. For example:
+
+```yaml
+dataset_inputs:
+    - { dataset: NMC }
+    - { dataset: ATLASTTBARTOT, cfac: [QCD] }
+    - { dataset: CMSZDIFF12, cfac: [QCD,NRM], sys: 10 }
+```
+
+We see that multiple datasets are inputted as a flat list and there is no
+hierarchy to the datasets, splitting them into experiments or process types.
+The grouping of datasets is done internally according to the metadata of
+datasets and is controlled by `metadata_group` key. This can be any key which
+is present in the `PLOTTING` file of each dataset - for example `experiment` or
+`nnpdf31_process`.
+
+If `metadata_group` is not specified in the runcard then it takes on the default
+value according to `data_grouping`. By default `data_grouping` is set to
+`standard_report` which corresponds `metadata_group` defaulting to `experiment`.
+
+For example the following runcard produces a single column table with a row containing
+the chi2 of the specificed datasets, grouped by `experiment`
+(the default grouping when nothing is specified).
+
+```yaml
+dataset_inputs:
+    - { dataset: NMC }
+    - { dataset: ATLASTTBARTOT, cfac: [QCD] }
+    - { dataset: CMSZDIFF12, cfac: [QCD,NRM], sys: 10 }
+
+theoryid: 53
+
+dataspecs:
+ - pdf: NNPDF31_nnlo_as_0118
+
+use_cuts: internal
+
+actions_:
+ - dataspecs_groups_chi2_table
+```
+
+If we add to the runcard to choose a different grouping:
+
+```yaml
+metadata_group: nnpdf31_process
+
+dataset_inputs:
+    - { dataset: NMC }
+    - { dataset: ATLASTTBARTOT, cfac: [QCD] }
+    - { dataset: CMSZDIFF12, cfac: [QCD,NRM], sys: 10 }
+
+theoryid: 53
+
+dataspecs:
+ - pdf: NNPDF31_nnlo_as_0118
+
+use_cuts: internal
+
+actions_:
+ - dataspecs_groups_chi2_table
+```
+
+then we instead get a single column table, but with the datasets grouped by
+process type, according the [theory uncertainties paper](https://arxiv.org/abs/1906.10698).
+
+## Backwards compatibility
+
+Most old validphys runcards which used the `experiments` key to specify a
+multi-levelled list of datasets should still work within the new framework. This
+is because if `dataset_inputs` is not present in the runcard, `validphys`
+attempts to find an `experiments` key and infer `dataset_inputs` from it.

doc/sphinx/source/vp/index.rst

@@ -13,5 +13,6 @@ vp-guide
    ./upload.md
    ./nnprofile.md
    ./scripts.md
+   ./dataspecification.md
    ./theorycov/index
    ./pydataobjs.rst

validphys2/src/validphys/app.py

@@ -42,6 +42,8 @@
              'validphys.theorycovariance.tests',
              'validphys.replica_selector',
              'validphys.closuretest',
+             # currently broken - will fix in NNPDF/nnpdf#511
+             # 'validphys.closure',
              'validphys.mc_gen_checks',
              'validphys.theoryinfo',
              'validphys.pseudodata',

validphys2/src/validphys/checks.py

@@ -118,9 +118,9 @@ def check_dataset_cuts_match_theorycovmat(dataset, fitthcovmat):
 
 
 @make_argcheck
-def check_experiment_cuts_match_theorycovmat(
-        experiment, fitthcovmat):
-    for dataset in experiment.datasets:
+def check_data_cuts_match_theorycovmat(
+        data, fitthcovmat):
+    for dataset in data.datasets:
         if fitthcovmat:
             ds_index = fitthcovmat.load().index.get_level_values(1)
             ncovmat = (ds_index == dataset.name).sum()

validphys2/src/validphys/comparefittemplates/report.md

@@ -42,9 +42,9 @@ Training validation
 -------------------
 {@fits plot_training_validation@}
 
-$\chi^2$ by experiment
+$\chi^2$ by group
 ----------------------
-{@plot_fits_experiments_chi2@}
+{@plot_fits_groups_data_chi2@}
 
 $\chi^2$ by dataset comparisons
 -------------------------------
@@ -53,11 +53,11 @@ $\chi^2$ by dataset comparisons
 ### Table
 {@fits_chi2_table(show_total=true)@}
 
-$\phi$ by experiment
+$\phi$ by group
 --------------------
-{@plot_fits_experiments_phi@}
+{@plot_fits_groups_data_phi@}
 
-Experiment plots
+Group plots
 ---------------
 {@with matched_datasets_from_dataspecs@}
 [Detailed plots for dataset ' {@dataset_name@} ']({@dataset_report report@})