Coffea 2023 basics questions

[Jailbone]

Hi there! I am exploring the new coffea 2023 since I liked the 0.7 version. I have some very basic questions:

1. First if all, what ist the use of a processor anymore? Since uproot now provides dask array creation?
2. I understand that the steps_per_file option now specifies the chunks when working with root files e.g.. But what is the purpose of the preprocess method? I have read somewhere that it also can be used for chunking. But I did not fully understand the concept or idea behind this functionality.
3. When specifying the chunks with steps_per_file I get a warning message (this should only be used for a small number of inputs...). I do not understand the reason it is giving me but in the end it this setting what speeds up my analysis. I guess it is because I only have one large file. So what to do when the input consists of many files? Not sure about all of that.

Thanks for some answers and I am looking forward to seeing the documentation grow!

[lgray]

1. Use of processor is no longer required, but we generally find that using a class or function to encapsulate your analysis is good for organization. We have included a new dataset_tools package that accepts old-style processors and anything that's callable as acceptable containers of analysis code.
2. steps_per_file generates optimized chunks in exactly two cases: when there is one input file, or when there are multiple files of exactly the same size. Neither of these well represent the reality of data provided a complete HEP analysis. When you have multiple large input datasets, you want to chunk up the files evenly asking for chunks of nearly the same size over all files in all datasets regardless of length. This is what preprocess does by calculating steps and specifying them per-file, in addition to a few other creature comforts. If you are using one file you may ignore this warning or filter it out using the standard python tools https://docs.python.org/3/library/warnings.html.
3. If you are analyzing one file you may ignore this warning. The warning is there to remind people that this doesn't scale well for typical HEP datasets.

[Jailbone]

Great overview, thank you very much for your effort, that helps! Some things I am wondering about in the second example:

Does the dataset definition have to follow this specific structure?
dataset = {"name": {"files": {"file1": {parameters}, ... }}, ...}
What does the steps parameter do here? Is there already some documentation of these parameters I can specify? I have not seen this till now.

Also there is no Processor involved in the second example, right? In the end the processing, so to speak, is happening through the apply_to_fileset method? Where can I learn about he available uproot options?

[lgray]

So the specification of that dataformat is given here - https://github.com/CoffeaTeam/coffea/blob/master/src/coffea/dataset_tools/preprocess.py#L126-L159

Where the input to preprocess is the Optional version of FilesetSpec, the "runnable" output of preprocess is the concrete FilesetSpec, i.e. all the fields are filled in so therefore unopenable files are dropped. The "updated" output may still have missing information, for instance if there was a file that could not be opened. In both outputs all glob strings are expanded to a concrete list of files.

There is also a forthcoming addition to preprocess that calculates further metadata about the files and saves it so that launching analysis on hundreds of datasets is fast.

Anyway - long story short: the input structure to preprocess has to look vaguely like { "name": { "files": <string or iterable>, ... }, ... } and the output structure will always be:

{ 
    "name": {
        "files": {
            "file1.root": {
                "object_path": "Events", 
                "steps": [[0, N], ... ],
                ...
            },
            ...
        },
        "metadata": { ... } # or None,
        "form": "..." # or None,
        ...
    },
    ...
}

It is json serializable so you can save the results and skip pre-processing later. If your input dataset is small enough / you're just doing testing, you probably don't need preprocess.

Concerning "processors" in the above example the analysis would be defined in my_analysis and apply_to_fileset repeatedly applies my_analysis to each list of files in each dataset, generating dictionary using the dataset names as keys and analysis data products (histograms, arrays, skims) as values. The apply_to_fileset function will take anything that is callable (has the __call__ operator defined or is a function) or inherits from ProcessorABC as the function to apply, your choice. We allow old-style processors since people have historical code at this point and it's easier to port the less you have to change.

You may learn about the uproot options here in the uproot documentation: https://uproot.readthedocs.io/en/latest/uproot._dask.dask.html (look lower on the page to see the information about specifying files to read)
and
https://github.com/scikit-hep/uproot5/blob/v5.2.1/src/uproot/_dask.py#L99-L102 for the failed-file report, which is missing from the published documentation for some reason (@jpivarski).

The failed-file report is poorly documented so far (it is very new, unless I missed something @douglasdavis). You can start to see how it works by looking into functions like https://github.com/CoffeaTeam/coffea/blob/master/src/coffea/dataset_tools/manipulations.py#L56-L134 that processes a dict of failed file reports into a FilesetSpec for reprocessing automatically. You can also make some fake datasets and see what it returns, I encourage playing around with it.

[douglasdavis]

Just to chime in on the failed-file report: yes it's currently undocumented, but it's something getting worked on here: dask-contrib/dask-awkward#444 and also planning on adding it to the main sphinx docs.

[nsmith-]

Just a comment on the example, there isn't any reason not to store the json with utf-8 encoding, i.e.

with gzip.open("normalized_data.json.gz", "wb") as f:
    f.write(bytes(json.dumps(dataset_runnable), encoding="ascii"))

could just as well be

with gzip.open("normalized_data.json.gz", "wt") as f:
    f.write(json.dumps(dataset_runnable))

[lgray]

Oh - there's no reason not to store in utf-8. Let the people have their emojis! Updated!

[lgray]

@Jailbone everything going OK so far after that reply?

[Jailbone]

Hey, yes thank you! I am playing around and testing things out. I sure will come back with new questions soon.