Enh/threshold metadata #189

bzah · 2022-06-17T13:57:47Z

Pull Request to resolve #183

Unit tests cover the changes.
These changes were tested on real data.
The relevant documentation has been added or updated.
A short description of the changes has been added to doc/source/references/release_notes.rst.

Describe the changes you made

icclim.index threshold parameter now properly update the index name and title in the output.
Add generic indices with rich metadata.
Add configurable Threshold
[breaking change] Make all the ECAD indices non configurable

The generic indices include:

CountOccurrences
for SU, TR, TG90p, TN90p, TX90p, FD, ID, TG10p, TN10p, TX10p, RR1, R10mm, R20mm, R75p, R95p, R99p, SD1, SD5cm, SD50cm, CD, CW, WD, WW
MaxConsecutiveOccurrence
for CSU, CFD, CDD, CWD
SumOfSpellLengths
WSDI, CSDI
Excess
GD4
Deficit
HD17
FractionOfTotal
R75pTOT, R95pTOT, r99pTOT
MeanOfDifference
DTR
DifferenceOfExtremes
ETR
MeanOfAbsoluteOneTimeStepDifference
vDTR
Maximum
RX1day, TXx, TNx, user_index - maximum
Minimum
TXn, TNn, user_index - minimum
Average
TG, TN, TX, SDII, SD, user_index - mean
Sum
PRCPTOT, user_index - sum
StandardDeviation
new
MaxOfRollingSum
RX5day, user_index - rolling_sum on extreme_mode=max
MinOfRollingSum
user_index - rolling_sum on extreme_mode=min
MaxOfRollingAverage
user_index - rolling_average on extreme_mode=max
MinOfRollingAverage
user_index - rolling_average on extreme_mode=min
DifferenceOfMeans
user_index - anomaly

Every generic function is configurable.
Those that depend on a threshold benefit from the Threshold class to allow to pass doy percentiles, period percentiles, per grid cells, scalar with unit and unit less scalar as a valid thresholds.
A few examples (syntax may change):

# count_occurrences
su_33 = icclim.index(in_files="data.nc", index_name="count_occurrences", threshold=">= 33 degC", slice_mode="jja")
wsdi_99 = icclim.index(in_files="data.nc", index_name="count_occurrences", threshold=">= 99 doy_per")
period_wsdi = icclim.index(in_files="data.nc", index_name="count_occurrences", threshold=">= 99 period_per")

# Excess
gd4_in_kelvin = icclim.index(in_files="data.nc", index_name="excess", threshold="277.15 K")
excess_of_pr = icclim.index(in_files="data.nc", index_name="excess", threshold="10 mmday")

# FractionOfTotal
r99pTOT = icclim.index(in_files="data.nc", index_name="fraction_of_total", threshold=Threshold(">= 99 period_per", min_value="1 mmday"))

Results of icclim.index are Dataset (or netcdf) with rich metadata including identifier, standard_name, long_name, description, cell_methods, short_name.
Example for su33 computed above:

short_name:       number_of_tx_days
standard_name:    number_of_days_when_maximum_air_temperature_greater_or_equal_to_33.0_degC
description:      Number of days of summer when maximum temperature is greater or equal to 33.0 degC.
long_name:        Number of days when tx is greater or equal to 33.0 degC.
units:            d
cell_methods:     time: sum over days

(note that standard_name should be changed to something more CF friendly).

- We can now compute a CDD or a heatwave We still can't compute CWD (`tn<{t1}` and `pr>{t2}`) though - when in_files is a dictionary, threshold can now be used the same way as `threshold` of `icclim.index`.

- Add registry class - remove ::compute from Indicator class (replaced by plain _call_ method) - turn IndexConfig into a dataclass - Make Threshold constructor handle percentile computation - Divide ClimateVar creation in two separate processes 1. When in_files is a dictionary 2. When in_files is a readable content directly (nc, zarr, da, ds) - BREAKING CHANGE: when in_files is a dict, renamed per_var_name to threshold_var_name

- `threshold` now works with strings thresholds It's either a threshold with unit (e.g. "22 degC") or doy percentiles (e.g. "75 doy_per") or period percentiles (e.g. "75 period_per") - `threshold` works with a combination such as ["1mm", "75 doy_per"] - Refactored read_climate_vars to always create a dictionary from the `var_names`

LogicalLink is not (yet?) part of icclim::index API.

coef is not part of icclim::index API, it can only be computed with a user_index.

date_event is now part of icclim::index API. It can be used on `maximum`, `minimum`, `max_of_rolling_average`, `max_of_rolling_sum`,`min_of_rolling_average`,`min_of_rolling_sum`.

I forgot to add it to count_occurrences (the first and last date of the occurrences, per sampling) and to max_consecutive_occurrence (start and end date of the spell). Also clean up dead code from user index.

This unit can be used with either `count_occurrences` or `fraction_of_total`. For ECAD indices, '%' is the default unit of rxxpTOT index family (as per specified in ATBD v11). Also fix some minor issues with date_event (explicit use of lat and lon dims).

For the Tier 1 and 2 variable, we should probably script the instantiation of StandardVariable. refs: - https://docs.google.com/spreadsheets/d/1qUauozwXkq7r1g-L4ALMIkCNINIhhCPx/edit#gid=1658589926 - https://www.wcrp-climate.org/wgcm-cmip/wgcm-cmip6 - https://cfconventions.org/Data/cf-standard-names/current/build/cf-standard-name-table.html

`::_guess_dataset_var_names` is capable of handling None standard_index. No need to raise an error.

They were unit tests and are now integration tests.

It was computing **min** instead of max.

Big bad commit of the day that refactor quite a few things following unit test update. - enh: Split `window_width` into 3 distinct parameter. - maint: renamed is_single_var into is_compared_to_reference for readability. - enh: Add `to_percent` to difference_of_means (aka anomaly). - enh: Improve metadata for unknown varaibles - fix: When adding a variable with `must_add_reference_var`, if time_range exist now it works. - maint: Simplified `must_add_reference_var` as `climate_vars_dict` is always a dictionary - enh: Improve input parsing for when the input is a list of mixed type (e.g. path to nc + a DataArray) - maint: update most tests to use generic indicator - maint: some more boring refactoring - maint: Add release notes

- Added `sampling_method` parameter to control the behavior of `slice_mode`. For now this is implemented only for `difference_of_means`.

github-actions · 2022-09-01T15:26:36Z

Report

File	Stmts	Miss	Cover	Missing
icclim
icclim_exceptions.py	7	1	86%	20
icclim_logger.py	85	29	66%	24, 43, 65, 70–95, 105, 111–142, 150–155
main.py	244	21	91%	101, 122, 131, 366, 451, 477–478, 480–481, 483, 485–486, 488–489, 508–515, 652, 659, 663
utils.py	32	3	91%	18, 29, 42
icclim/generic_indices
cf_var_metadata.py	50	1	98%	26
generic_indicators.py	461	79	83%	53, 56, 72, 76, 80, 97, 103, 202, 248, 255, 380, 707, 738, 744, 752, 771, 854, 874–882, 886, 916, 977–978, 1009–1036, 1063–1086, 1102, 1117–1147, 1153–1157
icclim/models
cf_calendar.py	35	1	97%	20
climate_variable.py	97	9	91%	92, 136, 147, 193, 199, 232, 260–261, 298
frequency.py	175	13	93%	167–170, 314–316, 323, 401, 427, 432, 483, 510
operator.py	29	1	97%	14
registry.py	30	1	97%	23
standard_index.py	30	2	93%	65, 69
threshold.py	134	30	78%	98–99, 104–117, 120, 143–147, 203–205, 219–230, 239–256, 272–273
icclim/pre_processing
input_parsing.py	186	46	75%	42, 103–118, 124–129, 139, 171, 187, 191, 201, 221, 230, 232, 239, 248, 276, 281–289, 309–325, 348
rechunk.py	87	2	98%	39, 129
icclim/tests
test_rechunk.py	27	2	93%	65, 93
testing_utils.py	22	2	91%	31, 44
TOTAL	2941	243	92%

Test results

Tests	Skipped	Failures	Errors	Time
143	0 💤	0 ❌	0 🔥	1m 42s ⏱️

bzah · 2022-09-01T15:28:52Z

Yipiti youpi, this pull request is now ready!

bzah · 2022-09-01T15:45:14Z

Before merge:

cleanup todos
update release note
pump coverage of generic_indicators.py to min 80%

Some indices were not using the expected operator. ">=" was used instead of ">" Plus the registry ::list was broken

It was broken when using the new groupby+resample sampling_method.

- Removed `identifier` from generic indicator in favor of `name`, which is **not** templated. - Fixed bootstrapped computation that could not run if "percentiles" was not in coords (because it was renamed) - Fixed some warnings

It should fix gh #73.

bzah added this to the 5.3 milestone Jun 17, 2022

bzah linked an issue Jun 17, 2022 that may be closed by this pull request

ENH: [metadata] Make tx90p configurable threshold appears in metadata #183

Closed

bzah force-pushed the enh/percentile_as_input branch from 2d8a841 to c59d370 Compare June 21, 2022 08:20

bzah force-pushed the enh/threshold_metadata branch from 09f2e7c to 3baf929 Compare June 21, 2022 08:46

Base automatically changed from enh/percentile_as_input to master June 21, 2022 08:48

bzah marked this pull request as ready for review June 21, 2022 08:48

This comment was marked as outdated.

Sign in to view

bzah requested a review from pagecp June 21, 2022 08:54

bzah marked this pull request as draft June 21, 2022 15:23

This comment was marked as outdated.

Sign in to view

bzah added 2 commits June 29, 2022 14:15

ENH: Make rxxp and rxxptot threshold configurable

af5434b

ENH: Update metadata when a custom threshold is used

f2dd6bb

bzah force-pushed the enh/threshold_metadata branch from 5b6c7a8 to 9b4498f Compare June 29, 2022 12:15

[skip ci] big wip again

052881d

bzah force-pushed the enh/threshold_metadata branch from 9b4498f to 052881d Compare June 30, 2022 21:07

it weurks

85f14f6

bzah mentioned this pull request Jul 6, 2022

IllegalMonthError for calculatng rx5day for a season from March until December for one year #180

Closed

bzah modified the milestones: 5.3, 6.0 Jul 6, 2022

bzah and others added 11 commits July 6, 2022 19:00

ENH: Make generic indices work on multiple inputs

79acf97

- We can now compute a CDD or a heatwave We still can't compute CWD (`tn<{t1}` and `pr>{t2}`) though - when in_files is a dictionary, threshold can now be used the same way as `threshold` of `icclim.index`.

Cleanup

510f85a

Merge remote-tracking branch 'origin/master' into enh/threshold_metadata

59f0213

MAINT: draft release notes

def38e4

[pre-commit.ci] auto fixes from pre-commit.com hooks

6e03908

ENH: clean output metadata

58ce894

ENH: Make it possible to compute CWD

bb335e0

[pre-commit.ci] auto fixes from pre-commit.com hooks

184c20a

ENH: Make it possible to compute CWD

6a55a66

bzah added 6 commits August 29, 2022 11:00

MAINT: Clean

c265d96

ENH: Re-add user index LogicalLink

287d386

LogicalLink is not (yet?) part of icclim::index API.

ENH: Re-add user index coef

2e1b707

coef is not part of icclim::index API, it can only be computed with a user_index.

ENH: Re-add date_event

94cd5ac

date_event is now part of icclim::index API. It can be used on `maximum`, `minimum`, `max_of_rolling_average`, `max_of_rolling_sum`,`min_of_rolling_average`,`min_of_rolling_sum`.

ENH: Re-add date_event (2)

66d35e6

I forgot to add it to count_occurrences (the first and last date of the occurrences, per sampling) and to max_consecutive_occurrence (start and end date of the spell). Also clean up dead code from user index.

MAINT: regenerate API

cb76220

bzah force-pushed the enh/threshold_metadata branch from cfd0d4c to cb76220 Compare August 29, 2022 17:04

bzah added 9 commits August 30, 2022 14:50

ENH: Add missing CF units to ecad indices

db1a579

ENH: Re-add '%' unit

72b24a3

This unit can be used with either `count_occurrences` or `fraction_of_total`. For ECAD indices, '%' is the default unit of rxxpTOT index family (as per specified in ATBD v11). Also fix some minor issues with date_event (explicit use of lat and lon dims).

MAINT: Remove useless error

e5207cb

`::_guess_dataset_var_names` is capable of handling None standard_index. No need to raise an error.

FIX: user_index mapping

1a64b09

MAINT: Update some user_index tests

803b930

They were unit tests and are now integration tests.

FIX: max_of_rolling_average

50cb49a

It was computing **min** instead of max.

FIX: DifferenceOfMeans Indicator (a.k.a. Anomaly)

37dbbe1

- Added `sampling_method` parameter to control the behavior of `slice_mode`. For now this is implemented only for `difference_of_means`.

[skip ci] DOC: Update coverage

de74418

bzah marked this pull request as ready for review September 1, 2022 15:28

bzah and others added 4 commits September 2, 2022 18:36

FIX: ECAD indices

6639036

Some indices were not using the expected operator. ">=" was used instead of ">" Plus the registry ::list was broken

FIX: difference_of_means

4d0a4fa

It was broken when using the new groupby+resample sampling_method.

MAINT: Cleanup and add integration tests

380940a

- Removed `identifier` from generic indicator in favor of `name`, which is **not** templated. - Fixed bootstrapped computation that could not run if "percentiles" was not in coords (because it was renamed) - Fixed some warnings

[skip ci] DOC: Update coverage

f603d5d

This was referenced Sep 2, 2022

ENH: Add documentation for generic indices #203

Closed

ENH: Add missing ECAD indices #204

Open

ENH: Make it possible to use % unit on fraction_of_total

a795078

It should fix gh #73.

bzah merged commit a34d39f into master Sep 2, 2022

bzah deleted the enh/threshold_metadata branch September 2, 2022 17:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enh/threshold metadata #189

Enh/threshold metadata #189

bzah commented Jun 17, 2022 •

edited

Loading

This comment was marked as outdated.

This comment was marked as outdated.

github-actions bot commented Sep 1, 2022 •

edited

Loading

bzah commented Sep 1, 2022

bzah commented Sep 1, 2022

Enh/threshold metadata #189

Enh/threshold metadata #189

Conversation

bzah commented Jun 17, 2022 • edited Loading

Pull Request to resolve #183

Describe the changes you made

This comment was marked as outdated.

This comment was marked as outdated.

github-actions bot commented Sep 1, 2022 • edited Loading

Test results

bzah commented Sep 1, 2022

bzah commented Sep 1, 2022

bzah commented Jun 17, 2022 •

edited

Loading

github-actions bot commented Sep 1, 2022 •

edited

Loading