Skip to content

Commit

Permalink
Adapt ESMValTool to new configuration (#3761)
Browse files Browse the repository at this point in the history
Co-authored-by: Bouwe Andela <b.andela@esciencecenter.nl>
  • Loading branch information
schlunma and bouweandela authored Oct 22, 2024
1 parent 5009b47 commit 8f7982c
Show file tree
Hide file tree
Showing 58 changed files with 462 additions and 545 deletions.
4 changes: 2 additions & 2 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -216,8 +216,8 @@ jobs:
conda activate esmvaltool
mkdir -p ~/climate_data
esmvaltool config get_config_user
echo "search_esgf: when_missing" >> ~/.esmvaltool/config-user.yml
cat ~/.esmvaltool/config-user.yml
echo "search_esgf: when_missing" >> ~/.config/esmvaltool/config-user.yml
cat ~/.config/esmvaltool/config-user.yml
for recipe in esmvaltool/recipes/testing/recipe_*.yml; do
esmvaltool run "$recipe"
done
Expand Down
274 changes: 0 additions & 274 deletions config-user-example.yml

This file was deleted.

9 changes: 5 additions & 4 deletions doc/sphinx/source/community/dataset.rst
Original file line number Diff line number Diff line change
Expand Up @@ -42,14 +42,15 @@ and run the recipe, to make sure the CMOR checks pass without warnings or errors

To test a pull request for a new CMORizer script:

#. Download the data following the instructions included in the script and place
it in the ``RAWOBS`` path specified in your ``config-user.yml``
#. Download the data following the instructions included in the script and
place it in the ``RAWOBS`` ``rootpath`` specified in your
:ref:`configuration <esmvalcore:config_options>`
#. If available, use the downloading script by running
``esmvaltool data download --config_file <config-file> <dataset>``
#. Run the cmorization by running ``esmvaltool data format <config-file> <dataset>``
#. Copy the resulting data to the ``OBS`` (for CMIP5 compliant data) or ``OBS6``
(for CMIP6 compliant data) path specified in your
``config-user.yml``
(for CMIP6 compliant data) ``rootpath`` specified in your
:ref:`configuration <esmvalcore:config_options>`
#. Run ``recipes/examples/recipe_check_obs.yml`` with the new dataset to check that
the data can be used

Expand Down
23 changes: 11 additions & 12 deletions doc/sphinx/source/community/diagnostic.rst
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ If it is just a few simple scripts or packaging is not possible (i.e. for NCL) y
and paste the source code into the ``esmvaltool/diag_scripts`` directory.

If you have existing code in a compiled language like
C, C++, or Fortran that you want to re-use, the recommended way to proceed is to add Python bindings and publish
C, C++, or Fortran that you want to reuse, the recommended way to proceed is to add Python bindings and publish
the package on PyPI so it can be installed as a Python dependency. You can then call the functions it provides
using a Python diagnostic.

Expand Down Expand Up @@ -134,9 +134,8 @@ Diagnostic output
Typically, diagnostic scripts create plots, but any other output such as e.g.
text files or tables is also possible.
Figures should be saved in the ``plot_dir``, either in both ``.pdf`` and
``.png`` format (preferred), or
respect the ``output_file_type`` specified in the
:ref:`esmvalcore:user configuration file`.
``.png`` format (preferred), or respect the :ref:`configuration option
<esmvalcore:config_options>` ``output_file_type`` .
Data should be saved in the ``work_dir``, preferably as a ``.nc``
(`NetCDF <https://www.unidata.ucar.edu/software/netcdf/>`__) file, following the
`CF-Conventions <https://cfconventions.org/>`__ as much as possible.
Expand Down Expand Up @@ -181,7 +180,7 @@ human inspection.
In addition to provenance information, a caption is also added to the plots.

Provenance information from the recipe is automatically recorded by ESMValCore, whereas
diagnostic scripts must include code specifically to record provenance. See below for
diagnostic scripts must include code specifically to record provenance. See below for
documentation of provenance attributes that can be included in a recipe.
When contributing a diagnostic, please make sure it records the provenance,
and that no warnings related to provenance are generated when running the recipe.
Expand Down Expand Up @@ -252,7 +251,7 @@ for example
plot_types:
errorbar: error bar plot
To use these items, include them in the provenance record dictionary in the form
To use these items, include them in the provenance record dictionary in the form
:code:`key: [value]`
i.e. for the example above as
:code:`'plot_types': ['errorbar']`.
Expand All @@ -275,8 +274,8 @@ Always use :func:`esmvaltool.diag_scripts.shared.run_diagnostic` at the end of y
with run_diagnostic() as config:
main(config)
Create a ``provenance_record`` for each diagnostic file (i.e. image or data
file) that the diagnostic script outputs. The ``provenance_record`` is a
Create a ``provenance_record`` for each diagnostic file (i.e. image or data
file) that the diagnostic script outputs. The ``provenance_record`` is a
dictionary of provenance items, for example:

.. code-block:: python
Expand All @@ -296,15 +295,15 @@ dictionary of provenance items, for example:
'statistics': ['mean'],
}
To save a matplotlib figure, use the convenience function
:func:`esmvaltool.diag_scripts.shared.save_figure`. Similarly, to save Iris cubes use
To save a matplotlib figure, use the convenience function
:func:`esmvaltool.diag_scripts.shared.save_figure`. Similarly, to save Iris cubes use
:func:`esmvaltool.diag_scripts.shared.save_data`. Both of these functions take
``provenance_record`` as an argument and log the provenance accordingly.
Have a look at the example Python diagnostic in
`esmvaltool/diag_scripts/examples/diagnostic.py <https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/diag_scripts/examples/diagnostic.py>`_
for a complete example.

For any other files created, you will need to make use of a
For any other files created, you will need to make use of a
:class:`esmvaltool.diag_scripts.shared.ProvenanceLogger` to log provenance. Include the
following code directly after the file is saved:

Expand Down Expand Up @@ -489,7 +488,7 @@ This includes the following items:
* In-code documentation (comments, docstrings)
* Code quality (e.g. no hardcoded pathnames)
* No Codacy errors reported
* Re-use of existing functions whenever possible
* Reuse of existing functions whenever possible
* Provenance implemented

Run recipe
Expand Down
Loading

0 comments on commit 8f7982c

Please sign in to comment.