Recipe test results for v2.10 #3463

bouweandela · 2023-12-12T11:31:11Z

Recipe test results for v2.10

Here is an overview of the tests done for releasing v2.10. The results are available in https://esmvaltool.dkrz.de/shared/esmvaltool/v2.10.0/debug.html.

Here is the conda environment.yml.

Recipe running session 2023-12-12

Recipes that ran successfully (133 out of 155)

recipe_albedolandcover.yml
recipe_anav13jclim.yml
recipe_arctic_ocean.yml
recipe_autoassess_landsurface_permafrost.yml
recipe_autoassess_landsurface_soilmoisture.yml
recipe_autoassess_landsurface_surfrad.yml
recipe_autoassess_stratosphere.yml
recipe_bock20jgr_fig_6-7.yml
recipe_bock20jgr_fig_8-10.yml
recipe_capacity_factor.yml
recipe_carvalhais14nat.yml
recipe_climwip_brunner2019_med.yml
recipe_climwip_brunner20esd.yml
recipe_climwip_test_basic.yml
recipe_climwip_test_performance_sigma.yml
recipe_clouds_ipcc.yml
recipe_cmug_h2o.yml
recipe_combined_indices.yml
recipe_concatenate_exps.yml
recipe_consecdrydays.yml
recipe_correlation.yml
recipe_cox18nature.yml
recipe_cvdp.yml
recipe_daily_era5.yml
recipe_deangelis15nat.yml
recipe_deangelis15nat_fig1_fast.yml
recipe_decadal.yml
recipe_diurnal_temperature_index.yml
recipe_eady_growth_rate.yml
recipe_ecs.yml
recipe_ecs_constraints.yml
recipe_ecs_scatter.yml
recipe_ensclus.yml
recipe_era5-land.yml
recipe_esacci_lst.yml
recipe_esacci_oc.yml
recipe_extract_shape.yml
recipe_extreme_events.yml
recipe_extreme_index.yml
recipe_flato13ipcc_figure_914.yml
recipe_flato13ipcc_figure_924.yml
recipe_flato13ipcc_figure_942.yml
recipe_flato13ipcc_figure_945a.yml
recipe_flato13ipcc_figure_96.yml
recipe_flato13ipcc_figure_98.yml
recipe_flato13ipcc_figures_926_927.yml
recipe_flato13ipcc_figures_92_95.yml
recipe_flato13ipcc_figures_938_941_cmip3.yml
recipe_flato13ipcc_figures_938_941_cmip6.yml
recipe_galytska23jgr.yml
recipe_gier2020bg.yml
recipe_globwat.yml
recipe_heatwaves_coldwaves.yml
recipe_hydro_forcing.yml
recipe_hyint.yml
recipe_hyint_extreme_events.yml
recipe_hype.yml
recipe_iht_toa.yml
recipe_impact.yml
recipe_ipccwg1ar6ch3_fig_3_43.yml
recipe_julia.yml
recipe_kcs.yml
recipe_lauer22jclim_fig1_clim.yml
recipe_lauer22jclim_fig1_clim_amip.yml
recipe_lauer22jclim_fig2_taylor.yml
recipe_lauer22jclim_fig2_taylor_amip.yml
recipe_lauer22jclim_fig6_interannual.yml
recipe_lauer22jclim_fig7_seas.yml
recipe_lauer22jclim_fig8_dyn.yml
recipe_lauer22jclim_fig9-11ab_scatter.yml
recipe_lauer22jclim_fig9-11c_pdf.yml
recipe_li17natcc.yml
recipe_lisflood.yml
recipe_marrmot.yml
recipe_martin18grl.yml
recipe_meehl20sciadv.yml
recipe_miles_block.yml
recipe_miles_eof.yml
recipe_miles_regimes.yml
recipe_modes_of_variability.yml
recipe_monitor.yml
recipe_monitor_with_refs.yml
recipe_mpqb_xch4.yml
recipe_multimodel_products.yml
recipe_my_personal_diagnostic.yml
recipe_ncl.yml
recipe_ocean_Landschuetzer2016.yml
recipe_ocean_amoc.yml
recipe_ocean_bgc.yml
recipe_ocean_example.yml
recipe_ocean_ice_extent.yml
recipe_ocean_multimap.yml
recipe_ocean_quadmap.yml
recipe_ocean_scalar_fields.yml
recipe_pcrglobwb.yml
recipe_perfmetrics_CMIP5.yml
recipe_perfmetrics_CMIP5_4cds.yml
recipe_perfmetrics_land_CMIP5.yml
recipe_preprocessor_derive_test.yml
recipe_preprocessor_test.yml
recipe_pv_capacity_factor.yml
recipe_python.yml
recipe_python_for_CI.yml
recipe_quantilebias.yml
recipe_r.yml
recipe_radiation_budget.yml
recipe_rainfarm.yml
recipe_runoff_et.yml
recipe_schlund20esd.yml
recipe_schlund20jgr_gpp_abs_rcp85.yml (run by @schlunma)
recipe_schlund20jgr_gpp_change_1pct.yml (run by @schlunma)
recipe_schlund20jgr_gpp_change_rcp85.yml (run by @schlunma)
recipe_sea_surface_salinity.yml
recipe_seaborn.yml
recipe_seaice.yml
recipe_seaice_feedback.yml
recipe_shapeselect.yml
recipe_smpi.yml
recipe_snowalbedo.yml
recipe_spei.yml
recipe_tcr.yml
recipe_thermodyn_diagtool.yml
recipe_toymodel.yml
recipe_validation.yml
recipe_validation_CMIP6.yml
recipe_variable_groups.yml
recipe_weigel21gmd_figures_13_16.yml
recipe_wenzel14jgr.yml
recipe_wenzel16nat.yml
recipe_wenzel16jclim.yml
recipe_williams09climdyn_CREM.yml
recipe_zmnam.yml

Recipes that failed because the diagnostic script failed (4 out of 155)

recipe_landcover.yml (fixed in Adapt to changed style scheme name in matplotlib #3475, due to a name change in matplotlib, see 3.6.0 and 3.8.0)
recipe_psyplot.yml (issue of a dependency, see IllegalArgumentException: CGAlgorithmsDD::orientationIndex encountered NaN/Inf numbers shapely/shapely#1921)
recipe_seaice_drift.yml (issue of a dependency, see IllegalArgumentException: CGAlgorithmsDD::orientationIndex encountered NaN/Inf numbers shapely/shapely#1921 and our own issue from last release recipe_seaice_drift.yml no longer runs #3243 )
recipe_russell18jgr.yml (see recipe_russell18jgr.yml fails to run #3478)

Recipes that failed because of missing data (4 out of 155)

recipe_bock20jgr_fig_1-4.yml
recipe_check_obs.yml
recipe_clouds_bias.yml
recipe_lauer13jclim.yml

Recipes that failed because the run took too long (8 out of 155)

recipe_eyring06jgr.yml
recipe_eyring13jgr_12.yml
recipe_ipccwg1ar6ch3_fig_3_19.yml
recipe_ipccwg1ar6ch3_fig_3_42_a.yml
recipe_ipccwg1ar6ch3_fig_3_42_b.yml
recipe_ipccwg1ar6ch3_fig_3_9.yml
recipe_lauer22jclim_fig3-4_zonal.yml
recipe_smpi_4cds.yml

Recipes that failed because they used too much memory (4 out of 155)

recipe_climate_change_hotspot.yml
recipe_lauer22jclim_fig5_lifrac.yml (tries to realize a 42GB array)
recipe_wflow.yml

Recipes that failed because of an HDF5 error (3 out of 155)

recipe_collins13ipcc.yml
recipe_ipccwg1ar6ch3_atmosphere.yml
recipe_tebaldi21esd.yml

The text was updated successfully, but these errors were encountered:

valeriupredoi · 2023-12-13T12:39:37Z

great stuff, folks! Here's me looking at the odd 17 ones that died of unnatural causes, of various reasons (not data or diag):

collins13 - the file /home/b/b381141/esmvaltool_output/esmvaltool-v2.10.x-2023-12-12/recipe_collins13ipcc_20231212_210804/preproc/IAV_calc_rtmt/rtmt/CMIP5_CNRM-CM5_Amon_piControl_r1i1p1_rtmt_1850-2699.nc that is causing a HDF5 issue is healthy, it is very possible a read was attempted while another process hadn't finished writing/closing it; a rerun should prob fix that
daily_era5 - it just died, prob node killed it for some personal reason
same as for easy_ipcc
same as for both eyrings
ipccAr6_atmosphere - file /home/b/b381141/esmvaltool_output/esmvaltool-v2.10.x-2023-12-12/recipe_ipccwg1ar6ch3_atmosphere_20231212_212329/preproc/IAV_calc_tas_cmip5/tas/CMIP5_CCSM4_Amon_piControl_r1i1p1_tas_0250-1300.nc is perfectly fine, prob the same issue as above for collins13
to be continued...

schlunma · 2023-12-13T14:07:52Z

recipe_psyplot fails due to an issue of geos, which will apparantly be fixed in their new version 3.13 (we currently have 3.11).

valeriupredoi · 2023-12-13T14:15:39Z

good find @schlunma 🍺 Our env is getting somewhat rather old, we need that Py312 support sooner than later (working on it for Core, stuck at prospector, the last hurdle)

valeriupredoi · 2023-12-13T14:35:27Z

lauer22_Fig5_lifrac fails to realize data from a (240, 91, 360, 720) array with dtype('float64') - needs be run on a hefty memory node

valeriupredoi · 2023-12-13T14:46:54Z

a lot of those 17 recipes, that failed with other reasons, simply die out - all goes fine until they just stop in their tracks - any info from the SLURM logs? It looks to me like they were just run on an interactive node and the user timed out, with the system killing the session, and implicitly, the running process

schlunma · 2023-12-13T15:07:22Z

I just successfully ran my 3 schlundjgr recipes, updated them in the table.

Regarding the NCL failures: Could you please cherry-pick 1647d46 into the release branch? This needs to be in there (see #3420). I just ran wenzel16jclim successfully with the current main branch.

bouweandela · 2023-12-13T20:49:38Z

Regarding the NCL failures: Could you please cherry-pick 1647d46 into the release branch? This needs to be in there (see #3420). I just ran wenzel16jclim successfully with the current main branch.

@schlunma I pulled this in, but it is giving me these issues:

$ cat /home/b/b381141/esmvaltool_output/esmvaltool-v2.10.x-2023-12-12/recipe_tebaldi21esd_20231213_193814/run/fig1b/plot_ts_line_mean_spread_pr/log.txt
 Copyright (C) 1995-2019 - All Rights Reserved
 University Corporation for Atmospheric Research
 NCAR Command Language Version 6.6.2
 The use of this software is governed by a License Agreement.
 See http://www.ncl.ucar.edu/ for more details.
INFO    Loading settings from /home/b/b381141/esmvaltool_output/esmvaltool-v2.10.x-2023-12-12/recipe_tebaldi21esd_20231213_193814/run/fig1b/plot_ts_line_mean_spread_pr/settings.ncl
INFO    Loading input data description from /home/b/b381141/esmvaltool_output/esmvaltool-v2.10.x-2023-12-12/recipe_tebaldi21esd_20231213_193814/preproc/fig1b/pr/pr_info.ncl
INFO     Wrote /home/b/b381141/esmvaltool_output/esmvaltool-v2.10.x-2023-12-12/recipe_tebaldi21esd_20231213_193814/plots/fig1b/plot_ts_line_mean_spread_pr//pr_ts_line_1850_2100.pdf
INFO    fatal: in log_provenance (interface_scripts/logging.ncl), outfile (path to figure) '/home/b/b381141/esmvaltool_output/esmvaltool-v2.10.x-2023-12-12/recipe_tebaldi21esd_20231213_193814/plots/fig1b/plot_ts_line_mean_spread_pr//pr_ts_line_1850_2100.pdf' does not exist (for PNGs, this function also searches for 'FILE.000001.png', 'FILE.000002.png', etc.); if no plot file is available use 'n/a'

and

$ cat /home/b/b381141/esmvaltool_output/esmvaltool-v2.10.x-2023-12-12/recipe_collins13ipcc_20231213_193815/run/ts_line_tas/ch12_plot_ts_line_mean_spread_tas/log.txt
 Copyright (C) 1995-2019 - All Rights Reserved
 University Corporation for Atmospheric Research
 NCAR Command Language Version 6.6.2
 The use of this software is governed by a License Agreement.
 See http://www.ncl.ucar.edu/ for more details.
INFO    Loading settings from /home/b/b381141/esmvaltool_output/esmvaltool-v2.10.x-2023-12-12/recipe_collins13ipcc_20231213_193815/run/ts_line_tas/ch12_plot_ts_line_mean_spread_tas/settings.ncl
INFO    Loading input data description from /home/b/b381141/esmvaltool_output/esmvaltool-v2.10.x-2023-12-12/recipe_collins13ipcc_20231213_193815/preproc/ts_line_tas/tas/tas_info.ncl
INFO     Wrote /home/b/b381141/esmvaltool_output/esmvaltool-v2.10.x-2023-12-12/recipe_collins13ipcc_20231213_193815/plots/ts_line_tas/ch12_plot_ts_line_mean_spread_tas//tas_ts_line_1850_2300.pdf
INFO    fatal: in log_provenance (interface_scripts/logging.ncl), outfile (path to figure) '/home/b/b381141/esmvaltool_output/esmvaltool-v2.10.x-2023-12-12/recipe_collins13ipcc_20231213_193815/plots/ts_line_tas/ch12_plot_ts_line_mean_spread_tas//tas_ts_line_1850_2300.pdf' does not exist (for PNGs, this function also searches for 'FILE.000001.png', 'FILE.000002.png', etc.); if no plot file is available use 'n/a'

with output_file_type: png in config-user.yml. The files with .png extension do exist, but somehow the code appears to be looking for pdf files?

bouweandela · 2023-12-14T09:10:14Z

@axel-lauer Could you please copy the file /work/bd0854/DATA/ESMValTool2/download/obs4MIPs/MODIS-1-0/v20180305/clt_mon_MODIS-1-0_BE_gn_200003-201109.nc to /work/bd0854/DATA/ESMValTool2/OBS/Tier1/MODIS-1-0/ on Levante? That will make it possible to run recipe_clouds_bias.yml and recipe_lauer13jclim again. Unfortunately, fully automatic download from ESGF does not work because the file has outdated facets on ESGF.

zklaus · 2023-12-14T09:39:27Z

The remaining NCL problems are due to a double guessing of the output filename. I think I fixed it locally, will commit soon.

zklaus · 2023-12-14T12:13:14Z

The NCL fix is in #3474.

valeriupredoi · 2023-12-14T15:12:26Z

can one of you pls have a look at Julia? I want to close (and will close) #3287 since the Julia looks is the only thing outstanding there (even so, it has an issue to it)

bouweandela · 2023-12-14T15:23:38Z

The issue with recipe_julia.yml is still open, I suspect it doesn't correctly handle fill values since the scale is at 1e20.

valeriupredoi · 2023-12-14T15:25:21Z

lemme have a look at Julia then 😁

valeriupredoi · 2023-12-14T16:49:40Z

ahaa! Figured out Julia! netCDF package loads missing values as 1e20s, whereas NCDatasets loads them as missing - will open Draft PR (I don't speak Julia so I couldn't make it work 100%)

bouweandela · 2023-12-15T09:21:00Z

Recipe running session 2023-12-14

I did a re-run of all recipes affected by late changes (i.e. all NCL recipes and a few other recipes with bug/data fixes).

Recipes that ran successfully (53 out of 155)

recipe_albedolandcover.yml
recipe_anav13jclim.yml
recipe_bock20jgr_fig_1-4.yml
recipe_bock20jgr_fig_6-7.yml
recipe_bock20jgr_fig_8-10.yml
recipe_climwip_brunner2019_med.yml
recipe_climwip_test_performance_sigma.yml
recipe_clouds_bias.yml
recipe_clouds_ipcc.yml
recipe_ecs_constraints.yml
recipe_ecs_scatter.yml
recipe_flato13ipcc_figure_924.yml
recipe_flato13ipcc_figure_945a.yml
recipe_flato13ipcc_figure_96.yml
recipe_flato13ipcc_figure_98.yml
recipe_flato13ipcc_figures_926_927.yml
recipe_flato13ipcc_figures_92_95.yml
recipe_flato13ipcc_figures_938_941_cmip3.yml
recipe_flato13ipcc_figures_938_941_cmip6.yml
recipe_galytska23jgr.yml
recipe_gier2020bg.yml
recipe_ipccwg1ar6ch3_fig_3_43.yml
recipe_ipccwg1ar6ch3_fig_3_9.yml
recipe_landcover.yml
recipe_lauer13jclim.yml
recipe_lauer22jclim_fig1_clim.yml
recipe_lauer22jclim_fig1_clim_amip.yml
recipe_lauer22jclim_fig2_taylor.yml
recipe_lauer22jclim_fig2_taylor_amip.yml
recipe_lauer22jclim_fig6_interannual.yml
recipe_lauer22jclim_fig7_seas.yml
recipe_lauer22jclim_fig8_dyn.yml
recipe_lauer22jclim_fig9-11ab_scatter.yml
recipe_lauer22jclim_fig9-11c_pdf.yml
recipe_miles_regimes.yml
recipe_monitor.yml
recipe_monitor_with_refs.yml
recipe_ncl.yml
recipe_perfmetrics_CMIP5.yml
recipe_perfmetrics_CMIP5_4cds.yml
recipe_perfmetrics_land_CMIP5.yml
recipe_radiation_budget.yml
recipe_schlund20esd.yml
recipe_seaice.yml
recipe_smpi.yml
recipe_smpi_4cds.yml
recipe_snowalbedo.yml
recipe_validation.yml
recipe_validation_CMIP6.yml
recipe_weigel21gmd_figures_13_16.yml
recipe_wenzel14jgr.yml
recipe_wenzel16jclim.yml
recipe_wenzel16nat.yml
recipe_wflow.yml

Recipes that failed because the diagnostic script failed (1 out of 155)

recipe_russell18jgr.yml (see recipe_russell18jgr.yml fails to run #3478; some diags do not write plots at all)

Recipes that failed because the run took too long (8 out of 155)

recipe_climate_change_hotspot.yml
recipe_eyring06jgr.yml
recipe_eyring13jgr_12.yml
recipe_ipccwg1ar6ch3_fig_3_19.yml
recipe_ipccwg1ar6ch3_fig_3_42_a.yml
recipe_ipccwg1ar6ch3_fig_3_42_b.yml
recipe_lauer22jclim_fig5_lifrac.yml

Recipes that failed because they used too much memory (1 out of 155)

recipe_lauer22jclim_fig3-4_zonal.yml (failed on node with 256GB RAM)

Recipes that failed with HDF5 errors (3 out of 155)

recipe_collins13ipcc.yml
recipe_ipccwg1ar6ch3_atmosphere.yml
recipe_tebaldi21esd.yml

axel-lauer · 2023-12-15T09:31:22Z

@axel-lauer Could you please copy the file /work/bd0854/DATA/ESMValTool2/download/obs4MIPs/MODIS-1-0/v20180305/clt_mon_MODIS-1-0_BE_gn_200003-201109.nc to /work/bd0854/DATA/ESMValTool2/OBS/Tier1/MODIS-1-0/ on Levante?

Done.

bouweandela · 2023-12-15T09:36:52Z

Some conclusions based on the above:

Recent changes to how NCL recipes record provenance have left 10 recipes broken (see Fixed plot paths in NCL provenance tracking #3422, Fix erroneous file_type handling in certain NCL diagnostics #3474)
The HD5 errors are reprodicible
I suspect that the longer run durations for some recipes are caused by a degradation in performance of the default dask scheduler

schlunma · 2023-12-15T09:38:38Z

Is the output of these tests available somewhere?

zklaus · 2023-12-15T09:42:39Z

Hm. I yesterday ran successfully two of the HDF5 problematic recipes (collins and tebaldi). I also ran successfully wenzel16jclim, though wenzel14jgr continues to fail also for me.

bouweandela · 2023-12-15T09:43:17Z

I have not uploaded it yet, but you can access it on Levante:
slurm logs: /home/b/b381141/esmvaltool-v2.10.x-2023-12-14-logs
esmvaltool output: /work/bd0854/b381141/esmvaltool_output/esmvaltool-v2.10.x-2023-12-12

bouweandela · 2023-12-15T09:54:55Z

Here is the HDF error if anyone is interested:
stdout:

OSError: [Errno -101] NetCDF: HDF error: '/home/b/b381141/esmvaltool_output/esmvaltool-v2.10.x-2023-12-12/recipe_tebaldi21esd_20231214_174448/preproc/fig6c_IAV/tas/CMIP6_MRI-ESM2-0_Amon_piControl_r1i1p1f1_tas_gn_1850-2150.nc'

stderr:

  #000: H5F.c line 836 in H5Fopen(): unable to synchronously open file
    major: File accessibility
    minor: Unable to open file
  #001: H5F.c line 796 in H5F__open_api_common(): unable to open file
    major: File accessibility
    minor: Unable to open file
  #002: H5VLcallback.c line 3863 in H5VL_file_open(): open failed
    major: Virtual Object Layer
    minor: Can't open object
  #003: H5VLcallback.c line 3675 in H5VL__file_open(): open failed
    major: Virtual Object Layer
    minor: Can't open object
  #004: H5VLnative_file.c line 128 in H5VL__native_file_open(): unable to open file
    major: File accessibility
    minor: Unable to open file
  #005: H5Fint.c line 1873 in H5F_open(): unable to lock the file
    major: File accessibility
    minor: Unable to lock file
  #006: H5FD.c line 2034 in H5FD_lock(): driver lock request failed
    major: Virtual File Layer
    minor: Unable to lock file
  #007: H5FDsec2.c line 988 in H5FD__sec2_lock(): unable to lock file, errno = 11, error message = 'Resource temporarily unavailable'
    major: Virtual File Layer
    minor: Unable to lock file

schlunma · 2023-12-15T10:03:36Z

I messed up the check for file existence in #3422, which gives these wrong NCL errors about non-existing plot files. Should be fixed by #3477 (I tested this successfully with wenzel14jgr, but since the other NCL errors are similar I guess those should run, too 🤞 )

bouweandela · 2023-12-15T10:17:53Z

Thanks, running them again now..

valeriupredoi · 2023-12-15T11:08:01Z

H5FDsec2.c line 988 in H5FD__sec2_lock(): unable to lock file, errno = 11, error message = 'Resource temporarily unavailable'

there is I/O toestepping going on - this is a fairly common HDF5 barf if the file is already opened by a process while another process is trying to open and read it/write to it - see eg h5py/h5py#1066 and a bunch of other people complaining about it since years ago. We need to understand what process opens the file and what other process is trying to do the same, but I think that's probably just on SLURM and that's not gonna be a straightforward task. At least, I'd be up for it but not today, and not before Xmas 🎄

schlunma · 2023-12-15T12:54:56Z

Good news, all NCL recipes with diag failures except for recipe_russel18jgr run now 🎉

recipe_russel18jgr fails since some of the diagnostic don't write plots (but judging from the code they are supposed to do that). This has already been the case for the v2.9.0 release, but became evident now due to the changes in the NCL provenance code. I opened an issue here: #3478

Since both maintainers of this recipe are not really active anymore, I suggest we flag this recipe as broken. @ESMValGroup/esmvaltool-coreteam opinions?

katjaweigel · 2023-12-15T13:39:30Z

It would be good, to get it running again, but I guess not worth/possible for the current release (since the missing figures were unnoticed/unreported for quite some time now).

schlunma · 2023-12-15T14:20:18Z

Here is a PR that fixes some issues in the russell recipe: #3479. With this, all diagnostics except for the ones listed in #3478 work again.

bouweandela · 2023-12-17T19:22:19Z

Here are the results from the comparison with v2.9.

bouweandela · 2023-12-17T20:40:19Z

Here is a summary of the comparison results (full comparison is here). @ESMValGroup/esmvaltool-recipe-maintainers and @ESMValGroup/esmvaltool-coreteam If you have a bit of time, please check if the output of these recipes is still correct. Tick the box and add your name behind a recipe once you've checked.

Runs with v2.10: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.10.0/
Runs with v2.9: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.9.0/
Runs with v2.8 https://esmvaltool.dkrz.de/shared/esmvaltool/v2.8.0/
If the plots or data files are not shown on the recipe output webpage (this happens when provenance has not been implemented in the diagnostic script), you can still download them by clicking the 'figures' or 'data' links at the bottom of the page.

The recipes where plots are different are probably the most important to check because if the data are different but the plots still look the same the changes are probably not significant. Maybe we can refine the thresholds for when data is reported as different for a future version of the comparison tool.

Plots and data are different

Only plots are different

Only data are different

Comparison is done using numpy.allclose with the default tolerances for floating point numbers and numpy.array_equal for other data types.

Results are the same as v2.9

recipe_anav13jclim.yml
recipe_autoassess_landsurface_permafrost.yml (but recipe appears to write provenance interface file to wrong directory)
recipe_autoassess_landsurface_surfrad.yml (but recipe appears to write provenance interface file to wrong directory)
recipe_autoassess_stratosphere.yml
recipe_capacity_factor.yml
recipe_carvalhais14nat.yml (compared to v2.8)
recipe_cmug_h2o.yml
recipe_combined_indices.yml
recipe_concatenate_exps.yml
recipe_consecdrydays.yml
recipe_correlation.yml
recipe_cox18nature.yml
recipe_cvdp.yml
recipe_deangelis15nat.yml
recipe_deangelis15nat_fig1_fast.yml
recipe_decadal.yml
recipe_diurnal_temperature_index.yml
recipe_ecs.yml
recipe_ensclus.yml
recipe_era5-land.yml
recipe_esacci_lst.yml
recipe_esacci_oc.yml
recipe_extract_shape.yml
recipe_extreme_events.yml
recipe_extreme_index.yml
recipe_flato13ipcc_figure_924.yml
recipe_flato13ipcc_figure_942.yml
recipe_flato13ipcc_figure_945a.yml
recipe_flato13ipcc_figure_96.yml
recipe_flato13ipcc_figures_926_927.yml
recipe_flato13ipcc_figures_938_941_cmip3.yml
recipe_globwat.yml
recipe_heatwaves_coldwaves.yml
recipe_hydro_forcing.yml
recipe_hyint.yml
recipe_hyint_extreme_events.yml
recipe_hype.yml
recipe_impact.yml
recipe_ipccwg1ar6ch3_fig_3_9.yml
recipe_julia.yml
recipe_landcover.yml
recipe_lauer22jclim_fig2_taylor.yml
recipe_lauer22jclim_fig2_taylor_amip.yml
recipe_lauer22jclim_fig6_interannual.yml
recipe_lauer22jclim_fig9-11ab_scatter.yml
recipe_lauer22jclim_fig9-11c_pdf.yml
recipe_li17natcc.yml
recipe_lisflood.yml
recipe_meehl20sciadv.yml
recipe_miles_block.yml
recipe_miles_eof.yml
recipe_miles_regimes.yml
recipe_my_personal_diagnostic.yml
recipe_ncl.yml
recipe_ocean_quadmap.yml
recipe_ocean_scalar_fields.yml
recipe_pcrglobwb.yml
recipe_preprocessor_derive_test.yml
recipe_preprocessor_test.yml
recipe_pv_capacity_factor.yml
recipe_python.yml
recipe_python_for_CI.yml
recipe_quantilebias.yml
recipe_r.yml
recipe_radiation_budget.yml
recipe_sea_surface_salinity.yml
recipe_seaice.yml
recipe_seaice_feedback.yml
recipe_snowalbedo.yml
recipe_tcr.yml
recipe_thermodyn_diagtool.yml
recipe_variable_groups.yml
recipe_weigel21gmd_figures_13_16.yml
recipe_wenzel14jgr.yml
recipe_wenzel16jclim.yml
recipe_wenzel16nat.yml
recipe_wflow.yml
recipe_williams09climdyn_CREM.yml
recipe_zmnam.yml

Unable to compare because no reference run for v2.9

recipe_easy_ipcc.yml (new recipe)
recipe_iht_toa.yml (new recipe)

katjaweigel · 2023-12-18T09:06:53Z

Is it possible to see the result/log of the comparison tool? Visually I don't find any difference in the figures of recipe_martin18grl.yml, but is has a lot of Figures, so I might have missed it.

bouweandela · 2023-12-18T09:29:46Z

Yes, they are posted in #3463 (comment).

recipe_martin18grl.yml: results differ from reference run
Reference run: /shared/esmvaltool/v2.9.0/recipe_martin18grl_20230704_162537
Current run: /shared/esmvaltool/v2.10.0/recipe_martin18grl_20231212_223414
Differing files:
  - plots/spi_collect/spi_collect/SPI_mapHistoric_Dur_of_Events_ACCESS1-0.png
  - plots/spi_collect/spi_collect/SPI_mapHistoric_Dur_of_Events_IPSL-CM5A-MR.png
  - plots/spi_collect/spi_collect/SPI_mapHistoric_Dur_of_Events_MPI-ESM-MR.png
  - plots/spi_collect/spi_collect/SPI_mapHistoric_No_of_Events_per_year_Observations.png
  - plots/spi_collect/spi_collect/SPI_mapHistoric_Sev_index_of_Events_GFDL-ESM2G.png
  - plots/spi_collect/spi_collect/SPI_mapObservations_Average_SPI_of_Events_Mean.png
  - plots/spi_collect2/spi_collect2/SPI_mapFuture_Avr_SPI_of_Events_GISS-E2-H.png
  - plots/spi_collect2/spi_collect2/SPI_mapFuture_Avr_SPI_of_Events_IPSL-CM5B-LR.png
  - plots/spi_collect2/spi_collect2/SPI_mapFuture_Dur_of_Events_CNRM-CM5.png
  - plots/spi_collect2/spi_collect2/SPI_mapFuture_Dur_of_Events_GFDL-ESM2G.png
  - plots/spi_collect2/spi_collect2/SPI_mapFuture_Dur_of_Events_HadGEM2-CC.png
  - plots/spi_collect2/spi_collect2/SPI_mapFuture_Dur_of_Events_IPSL-CM5A-LR.png
  - plots/spi_collect2/spi_collect2/SPI_mapFuture_Dur_of_Events_MPI-ESM-MR.png
  - plots/spi_collect2/spi_collect2/SPI_mapFuture_Dur_of_Events_MRI-ESM1.png
  - plots/spi_collect2/spi_collect2/SPI_mapFuture_Sev_index_of_Events_GFDL-ESM2G.png
  - plots/spi_collect2/spi_collect2/SPI_mapFuture_Sev_index_of_Events_IPSL-CM5B-LR.png
  - plots/spi_collect2/spi_collect2/SPI_mapFuture_Sev_index_of_Events_MRI-ESM1.png
  - plots/spi_collect2/spi_collect2/SPI_mapHistoric_Dur_of_Events_GFDL-ESM2G.png
  - plots/spi_collect2/spi_collect2/SPI_mapHistoric_Dur_of_Events_GISS-E2-H.png
  - plots/spi_collect2/spi_collect2/SPI_mapHistoric_Sev_index_of_Events_GFDL-CM3.png
  - plots/spi_collect2/spi_collect2/SPI_mapHistoric_Sev_index_of_Events_MRI-ESM1.png
  - plots/spi_collect2/spi_collect2/SPI_mapHistoric_Sev_index_of_Events_NorESM1-M.png

Thanks for checking!

katjaweigel · 2023-12-18T09:47:15Z

Thanks! (And sorry that I didn't get the idea to look at the post before.)
The differences are tiny deviation in the way missing data are masked on the plot (really only visible, if switch between the two versions of these figures.)

valeriupredoi · 2023-12-18T14:53:09Z

autoassess, validation, and all @ledm 's oceans eleven look fine! Stellar work @bouweandela 🍺

bouweandela · 2023-12-20T09:53:21Z

Thanks, everyone! The release has now been published!

valeriupredoi · 2023-12-20T09:54:59Z

It's a Christmas miracle 🎄 🎅

bouweandela added the release label Dec 12, 2023

This comment was marked as outdated.

Sign in to view

bouweandela mentioned this issue Dec 13, 2023

Recipe testing and output comparison for release 2.10.0 #3414

Closed

valeriupredoi mentioned this issue Dec 14, 2023

Recipes with issues discovered during the 2.8.0 release runs #3140

Closed

valeriupredoi mentioned this issue Dec 14, 2023

Recipes with issues discovered during the 2.9.0 release runs #3287

Open

schlunma mentioned this issue Dec 15, 2023

Fix NCL provenance tracking #3477

Merged

12 tasks

This was referenced Dec 19, 2023

recipe_psyplot.yml does not run with shapely 2 #3483

Closed

Recipes with issues discovered during the 2.10.0 release test runs #3484

Open

bouweandela closed this as completed Dec 20, 2023

chrisbillowsMO mentioned this issue May 16, 2024

Recipe test results for ESMValCore v2.11.0rc1 ESMValGroup/ESMValCore#2421

Closed

mo-gill mentioned this issue Jun 13, 2024

Recipe testing and comparison for release 2.11.0 #3616

Closed

ehogan mentioned this issue Jul 1, 2024

[v2.11.0 release] Recipe failures on DKRZ: OSError: [Errno -101] NetCDF: HDF error #3702

Open

Recipe test results for v2.10 #3463

Recipe test results for v2.10 #3463

Comments

bouweandela commented Dec 12, 2023 • edited by schlunma Loading

Recipe test results for v2.10

Recipe running session 2023-12-12

Recipes that ran successfully (133 out of 155)

Recipes that failed because the diagnostic script failed (4 out of 155)

Recipes that failed because of missing data (4 out of 155)

Recipes that failed because the run took too long (8 out of 155)

Recipes that failed because they used too much memory (4 out of 155)

Recipes that failed because of an HDF5 error (3 out of 155)

This comment was marked as outdated.

valeriupredoi commented Dec 13, 2023 • edited Loading

schlunma commented Dec 13, 2023

valeriupredoi commented Dec 13, 2023

valeriupredoi commented Dec 13, 2023

valeriupredoi commented Dec 13, 2023

schlunma commented Dec 13, 2023

bouweandela commented Dec 13, 2023 • edited Loading

bouweandela commented Dec 14, 2023 • edited Loading

zklaus commented Dec 14, 2023

zklaus commented Dec 14, 2023

valeriupredoi commented Dec 14, 2023

bouweandela commented Dec 14, 2023

valeriupredoi commented Dec 14, 2023

valeriupredoi commented Dec 14, 2023

bouweandela commented Dec 15, 2023 • edited by schlunma Loading

Recipe running session 2023-12-14

Recipes that ran successfully (53 out of 155)

Recipes that failed because the diagnostic script failed (1 out of 155)

Recipes that failed because the run took too long (8 out of 155)

Recipes that failed because they used too much memory (1 out of 155)

Recipes that failed with HDF5 errors (3 out of 155)

axel-lauer commented Dec 15, 2023

bouweandela commented Dec 15, 2023

schlunma commented Dec 15, 2023

zklaus commented Dec 15, 2023

bouweandela commented Dec 15, 2023

bouweandela commented Dec 15, 2023

schlunma commented Dec 15, 2023

bouweandela commented Dec 15, 2023

valeriupredoi commented Dec 15, 2023

schlunma commented Dec 15, 2023 • edited Loading

katjaweigel commented Dec 15, 2023

schlunma commented Dec 15, 2023

bouweandela commented Dec 17, 2023

bouweandela commented Dec 17, 2023 • edited by schlunma Loading

Plots and data are different

Only plots are different

Only data are different

Results are the same as v2.9

Unable to compare because no reference run for v2.9

katjaweigel commented Dec 18, 2023

bouweandela commented Dec 18, 2023 • edited Loading

katjaweigel commented Dec 18, 2023

valeriupredoi commented Dec 18, 2023

bouweandela commented Dec 20, 2023

valeriupredoi commented Dec 20, 2023

bouweandela commented Dec 12, 2023 •

edited by schlunma

Loading

valeriupredoi commented Dec 13, 2023 •

edited

Loading

bouweandela commented Dec 13, 2023 •

edited

Loading

bouweandela commented Dec 14, 2023 •

edited

Loading

bouweandela commented Dec 15, 2023 •

edited by schlunma

Loading

schlunma commented Dec 15, 2023 •

edited

Loading

bouweandela commented Dec 17, 2023 •

edited by schlunma

Loading

bouweandela commented Dec 18, 2023 •

edited

Loading