consolidate dataclasses

[lukasheinrich]

continue discussion from #1554 cc @lhenkelm @alexander-held @matthewfeickert

there seems to be a need for container objects/dataclasses/tuples to carry around some details of internal things

examples:

OptimizerResult
FitResult (from cabinetry)
HypoTestResult (like the one from ROOT)
HypoTestScan (for CLs plots)
CalculatorDetails (like BestFitParameters from @lhenkelm in #1554)

others could even be

FitConfig (S+B configuratiion, B-only configuration, like ROOT or our (init_pars,par_bounds,fixed_pars) tuple)
Data (like ROOT's RooDataset)

this could be designed in a rather pyhf-independent way (like tensorlib)

this discussin should try to tackle

• what is the list of required data classes
• (how do they mesh with diffability / tensor backends)
• where should they live (their own repo?)

[kratsg]

I would not mind if we started a src/pyhf/models here for the dataclasses needed. And then upstream toolings like @alexander-held with cabinetry can inherit to extend as needed. Or at least say this is compatible -- but then this is just a typing exercise at this point. Dataclasses are just named tuples with specific attributes/APIs. So the question is just about having this documented explicitly/typed. So is this about typing?

[alexander-held]

I will try to provide a bit of information about the status in cabinetry here.

The containers used in cabinetry were created for the purpose of grouping together related quantities in an easy-to-handle format, especially for visualization purposes. The basic idea is that the object returned from some inference step contains what is needed to visualize the results, or to otherwise use them to produce other output:

fit_results = cabinetry.fit.fit(model, data)

cabinetry.visualize.pulls(fit_results)  # plot needs best-fit parameters and uncertainties
cabinetry.visualize.correlation_matrix(fit_results)  # plot needs parameter correlations

# model prediction needs best-fit parameters, uncertainties, correlations
model_prediction = cabinetry.model_utils.prediction(model, fit_results=fit_results)

This approach means that users do not need to think about all the details about which function needs which aspects of a MLE result, but they can access individual parts of it if needed. cabinetry currently assumes that parameter uncertainties and correlations are available. Supporting minimizers who don't provide this (and the results object they return) would require some additions to catch these cases.

Since the containers were written with the cabinetry API in mind, the objects contained within are essentially all there because they are needed somewhere else in the API. Other objects that may also fit into one of the existing containers but are not needed elsewhere are generally not included at the moment.

You can also find the containers in the code / documentation. They are all simple named tuples at the moment.

Here is a list of the current containers used in cabinetry:

• FitResults, the results from a MLE: best-fit parameters, uncertainties, parameter labels, correlation matrix, NLL, goodness-of-fit
• RankingResults, results from nuisance parameter ranking by impact: best-fit parameters and uncertainties (from MLE), parameter labels, lists of pre-/post-fit up/down impact
• ScanResults, results from a likelihood scan: parameter name and best-fit value/uncertainty (to draw Gaussian approximation), then list of probed parameter values and list of corresponding NLLs
• LimitResults, results from an upper parameter limit calculation: observed limit (float), expected limit (list including 1/2 sigma bands), list of POI values probed, and for each POI value the corresponding observed CLs and list of expected CLs values (including bands)
• SignificanceResults, results from discovery significance calculation: this one is not actually used for anything else, observed/expected significance and p-value

There is also a fairly new ModelPrediction (documentation) which carries around a yield prediction, but that is probably not of interest to pyhf.

All the containers in cabinetry are fairly high-level. I think a system of inheriting from a pyhf base class and extending where needed sounds quite promising.

[lhenkelm]

IMO dataclasses are often useful (just as named tuples are) because they communicate composite return values clearly, in a static way.
This makes it more difficult to wrongly unpack return values, and makes the code easier to read. So I like the suggestion
(and I agree with all of @alexander-held s good points about having custom containers).

Regarding where to put them, I dont think this has to be pre-planned big hairy project.
Dataclasses (or namedtuples where implicit unpacking ala one, two, three = returns_triple() is wanted) can just incrementally replace complex return values (or sets of args that always travel together) as the project evolves & the need becomes clear. With a centrally defined set of dataclasses I see a risk of functions being written to match these prescribed types, and to me that puts the cart before the horse, the types should follow from the needs of the functions. So in my (personal & outsider) opinion, the best place for namedtuple/dataclass definitions is right next to the functions that return their instances.

Regarding where dataclasses might be good to have, @lukasheinrich has many good suggestions. Especially useful from a user perspective:

• infer.mle.fit and infer.mle.fixed_poi_fit could return a type that always provides access to all information the fit implementation allows. I.e. a ScipyFitResult or a MinuitFitResult, depending on the backend.
• infer.hypotest has a rough screen's worth of docs describing the structure of the returned tuple and how it depends on the arguments. A dataclass won't make the docs shorter, but it will make it much more difficult to accidentally unpack something wrongly, especially for new users. (+ the need to have so many "return_" kwargs disappears)
• infer.upperlimit is less complicated, but the length of the return value still depends on the args, and unpacking expected limits into observed and vice versa is a nasty silent error to have.

As far as harmonizing with cabinetry goes, I think this is a great idea in terms of making sure that the same attributes are bound to the same names, the classes group together similar attributes etc. to make duck-typing work where possible.
However subclassing does introduce a very strong coupling.
Writing a factory method in cabinetry.FitResults that takes a (hypothetical) pyhf.MinuitOptimizerResult and a pyhf.Model is almost as convenient as extending the type (just call FitResults.from_pyhf(bestfit, model)). And it leaves either project free to change things up, as all that needs updating is the factory method.