cookbook with recipes in main repo

[wsavran]

@mherrmann3 @bayonato89 @pabloitu we can continue the discussion from #119 in this issue. we have started a notebooks folder in the main repo that will contain cookbooks and recipes to perform useful tasks using pycsep (and other python scientific computing libraries).

this is an open discussion to brainstorm some items that should be included in this cookbook. ideally, some of these things might already exist from our work.

[mherrmann3]

To summarize and generalize the cookbook idea, we could host recipes as extensions/addons to pycsep, i.e., optional functionalities that build up on (or are related to) pycsep, but that are too specific to be part of pycsep itself, that is, its core competence. These extensions could be:

• new kinds of tests or scores using pycsep's framework (incl. their graphical representation)
  • like Added a function that computes log-likelihood scores #118 and Added a function that plots log-likelihood scores #119 as discussed therin
• alternative visualizations of forecast comparisons, or of models.EvaluationResult for existing tests
• minor plotting tools (time vs. magnitude, time vs. mean forecast rate, etc.)
• import/export of alternative catalog/forecast formats, i.e., readers and writers
• description of additional or alternative workflows
• ...?

In this way, we are more open to new contributions from anyone, potentially making it easier and faster to use and extend pycsep. pycsep itself will remain clean and focused, instead of becoming an increasingly harder-to-maintain multitool. We may consider integrating an extension into pycsep's core if it's deemed very popular/useful.

Needs clarification

The current plan is that recipes are shipped with pycsep via the 'cookbooks' in docs, i.e., one needs to copy-paste their codes to use them.

• How we want to represent those? Each extension as a separate notebook? A notebook collecting all-in-one making use of subsections?
• How to maintain them? What if the API of pycsep or of other dependent packages changes? We should at least state the pycsep version for which a recipe is compatible.
• Regarding a small accompanying documentation of a recipe, are in-code comments sufficient (like a gist) or do we want a separate description?
• Which contributions should be considered as extensions, and which as improvement of pycsep's core?

Alternatively, we could interface them in pycsep, e.g., pycsep.extensions.<extension>. In this way, even if one extension is popular, we won't have to integrate it into pycsep, as its are already usable via API. However, this makes their maintenance hardly easier than integrating them in pycsep right away (and may also make pycsep dependent on more packages). Two solutions:

• we openly claim that they are not officially supported and that functionality may break
• maintain them separately (i.e. outside pycsep) - likely an overkill (for now)

Possible candidates

• plot_log_likelihood_scores / plot_diff_spatial_likelihood of Added a function that plots log-likelihood scores #119, which actually kick-started the idea of @wsavran.

Some further candidates could be functions that already exist but are currently undocumented:

• csep\core\poisson_evaluations.py: poisson_spatial_likelihood and binary_spatial_likelihood (related to Added a function that plots log-likelihood scores #119 as mentioned above)
• csep\core\regions.py: {italy|california}_relm_collection_region
• csep\utils\plots.py: plot_cumulative_events_versus_time_dev, an alternative for plot_cumulative_events_versus_time
• csep\utils\plots.py: plot_magnitude_histogram_dev, an alternative for plot_magnitude_histogram

@wsavran mentioned the last two already in #119. So we could extend the outsourcing also to functions that are documented, but not part of pycsep's core, such as:

• csep\utils\plots.py: plot_magnitude_versus_time
• csep\utils\plots.py: plot_ecdf
• csep\utils\plots.py: add_labels_for_publication

Btw

I also found some currently undocumented & unused utility functions; maybe they can be pruned away:

• csep\core\regions.py: grid_spacing
• csep\utils\calc.py: find_nearest
• csep\utils\calc.py: func_inverse
• csep\utils\calc.py: _compute_approximate_likelihood
• csep\utils\calc.py: _compute_spatial_statistic
• csep\utils\calc.py: _distribution_test

[wsavran]

The distinction you make about extensions vs. recipes makes a lot of sense, because they serve two different purposes. The former would contain working functions that users might want to use directly in their code. For example, csep\utils\plots.py: plot_magnitude_versus_time or csep\utils\plots.py: add_labels_for_publication or the functions that @bayonato89 wrote. The latter would contain examples on how to do various things with pyCSEP. An example of this might be, how to create a catalog from a pandas DataFrame. Or how to scale a forecast. The list goes on...

I like the csep.extensions.* idea for storing the functionality we'd want to distribute new with pyCSEP releases, but not actually including as part of the main toolkit. Like you say, we should make it clear that these are not seeing on-going support. We can test this out with the refactoring in an upcoming release (target for v0.7.0).

The recipes/cookbook should probably be part of the documentation following the lead from matplotlib.

For now, we should leave the notebooks folder in place, because at some point I'd like to distribute some notebooks that are complete for users to get started. They could represent a mini-experiment that might plot multiple evaluations and forecasts. For instance, the user would modify a couple of file paths and metadata about their forecasts, but the notebook would produce a set of evaluations.