Feature/dei 35 multiply rule time dependent table (#32)

* Feat [DEI-96]: working date range table

* feat[DEI-35]: fix date_range not passed along to rule

* feat [DEI-35]: update tests

* feat [DEI-35]: test validation utils

* feat [DEI-35]: update tests multiply rule

* feat [DEI-35]: update parser multiply  rule tests

* feat [EIT-35]: add validation for start_date before end_date

* feat [DEI-35]: fix flake8

* feat [DEI-35]: fix flake8 again

* feat [DEI-35]: and again flake8

* feat [DEI-35]: update from review feedback

* feat [DEI-35]: update mkdocs

* fix feedback comments

* Added functionality for REACT to calculate Q and depth from WFLOW

Added functionality for REACT to calculate Q and depth from WFLOW (this is a test application of the functionality but as the WFLOW file seems to contain lakes I don't think this approach will give a correct result -> approach needs to be refactured).

---------

Co-authored-by: Marc Weeber <41464989+MPWeeber@users.noreply.github.com>

batch_run_examples.bat

@@ -0,0 +1,78 @@
+@echo "Run the tests"
+python examples/python_test_of_functions.py
+python main.py examples/test_DHYDRO_DELWAQ_Online.yaml
+python main.py examples/test_DHYDRO_Maas.yaml
+python main.py examples/test_DHYDRO_Maas_Potamogeton_testcase_fuzzylogic.yaml
+python main.py examples/test_IMOD_Dommel.yaml
+@echo next one is to be considered requested functionality (different time axes)
+@echo 2023-05-31 10:51:46,661: ValueError: The arrays must have the same dimensions.
+@echo python main.py examples/test_SFINCS_beira_flood.yaml
+@echo 
+python main.py examples/test_VKZM_Potamogeton_testcase_boundaries.yaml
+python main.py examples/test_VKZM_Potamogeton_testcase_fuzzylogic.yaml
+python main.py examples/test_VKZM_simple_testcase.yaml
+python main.py examples/test_IRM_Doesburg_IJssel_rasters.yaml
+python main.py examples/test_Westerschelde.yaml
+python main.py examples/test1_default_independent_rules.yaml
+python main.py examples/test2_variable_mapping.yaml
+@echo next one is to be considered requested functionality  (error in yaml implementation, always variable mapping expected)
+@echo AttributeError: 'str' object has no attribute 'keys'
+@echo python main.py examples/test2b_variable_no_mapping.yaml
+@echo
+@echo next one is to be considered requested functionality  (error in yaml implementation, always variable mapping expected)
+@echo 2023-05-31 10:53:25,352: ERROR    Mapping towards the following variables 'mesh2d_sa1, mesh2d_s1', will create duplicates with variables in the input datasets.
+@echo 2023-05-31 10:53:25,352: ERROR    Model "Model 1" transition from ModelStatus.VALIDATING to ModelStatus.VALIDATED has failed.
+@echo python main.py examples/test2c_variable_no_mapping.yaml
+@echo 
+python main.py examples/test3_layer_filter_layer22.yaml
+python main.py examples/test3b_layer_filter_layer0.yaml
+python main.py examples/test3c_layer_filter_layer1.yaml
+@echo 
+@echo next one is correct (expected error in model config,  not existing layer) WARNING could be clearer
+@echo 2023-05-31 13:34:14,487: WARNING  Some rules can not be resolved: test name
+@echo 2023-05-31 13:34:14,487: ERROR    Initialization failed.
+@echo python main.py examples/test3d_layer_filter_not_existing_layer.yaml
+@echo 
+python main.py examples/test4_depending_rules_multiply.yaml
+python main.py examples/test4b_depending_rules_multiply.yaml
+python main.py examples/test5_depending_rules_multiply_layerfilter.yaml
+python main.py examples/test6_combine_results_add.yaml
+python main.py examples/test6b_combine_results_subtract.yaml
+python main.py examples/test6c_combine_results_min.yaml
+python main.py examples/test6d_combine_results_max.yaml
+python main.py examples/test6e_combine_results_multiply.yaml
+python main.py examples/test6f_combine_results_with_nans.yaml
+python main.py examples/test7_time_aggregation_year.yaml
+python main.py examples/test7b_time_aggregation_month.yaml
+@echo next one is correct (expected error in model config, day is not implemented)
+@echo 2023-05-31 10:54:04,909: ERROR    The provided time scale 'day' of rule 'test name' is not supported.
+@echo Please select one of the following types: month,year
+@echo python main.py examples/test7c_time_aggregation_day_not_existing_opp.yaml
+@echo 
+@echo next one is correct (expected error in model config, popcorn is not implemented)
+@echo The provided time scale 'popcorn' of rule 'test name' is not supported.
+@echo Please select one of the following types: month,year
+@echo python main.py examples/test7d_time_aggregation_popcorn_not_existing_opp.yaml
+@echo 
+@echo next one is correct (expected error in model config,  time scale missing)
+@echo AttributeError: Missing element time_scale
+@echo python main.py examples/test7e_time_aggregation_default.yaml
+@echo 
+python main.py examples/test8_stepwise_classification_waterlevel.yaml
+python main.py examples/test8b_stepwise_classification_salinity_limits.yaml
+python main.py examples/test8c_stepwise_classification_salinity_categories.yaml
+python main.py examples/test8d_stepwise_classification_waterlevel_lower_range.yaml
+python main.py examples/test9_response_curve_waterlevel.yaml
+@echo next one is correct (expected error in model config, data provided in wrong order)
+@echo 2023-05-31 10:57:54,424: ERROR    The input values should be given in a sorted order.
+@echo 2023-05-31 10:57:54,424: ERROR    Model "Model 1" transition from ModelStatus.VALIDATING to ModelStatus.VALIDATED has failed.
+@echo python main.py examples/test9b_response_curve_waterlevel_wrong_order.yaml
+@echo 
+python main.py examples/test9c_response_curve_waterlevel_on_year.yaml
+python main.py examples/test10_formula_based_comparison.yaml
+python main.py examples/test10b_formula_based_arithmatic.yaml
+python main.py examples/test10c_formula_based_ifelse.yaml
+@echo next one is to be considered requested functionality (variable computations with missing/different time axes) 
+@echo IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed
+@echo python main.py examples/test10d_formula_based_timeaxes_calculation.yaml
+@echo 

decoimpact/business/entities/rules/multiply_rule.py

@@ -7,6 +7,8 @@
 
 from typing import List
 
+from datetime import datetime as _dt
+import numpy as _np
 import xarray as _xr
 
 from decoimpact.business.entities.rules.i_array_based_rule import IArrayBasedRule
@@ -21,29 +23,51 @@ def __init__(
         self,
         name: str,
         input_variable_names: List[str],
-        multipliers: List[float],
+        multipliers: List[List[float]],
         output_variable_name: str = "output",
+        date_range: List[List[str]] = [],
         description: str = "",
     ):
         super().__init__(name, input_variable_names, output_variable_name, description)
         self._multipliers = multipliers
+        self._date_range = date_range
 
     @property
-    def multipliers(self) -> List[float]:
+    def multipliers(self) -> List[List[float]]:
         """Multiplier property"""
         return self._multipliers
 
-    def execute(self, value_array: _xr.DataArray, logger: ILogger) -> _xr.DataArray:
+    @property
+    def date_range(self) -> List[List[str]]:
+        """Date range property"""
+        return self._date_range
 
-        """Multiplies the value with the specified multipliers
+    def execute(self, value_array: _xr.DataArray, logger: ILogger) -> _xr.DataArray:
+        """Multiplies the value with the specified multipliers. If there is no
+        date range, multiply the whole DataArray with the same multiplier. If
+        there is table format with date range, make sure that the correct values
+        in time are multiplied with the corresponding multipliers.
         Args:
-            value_array (DataArray): values to multiply
+            value_array (DataArray): Values to multiply
         Returns:
             DataArray: Multiplied values
         """
+        # Per time period multiple multipliers can be given, reduce this to
+        # one multiplier by taking the product of all multipliers.
+        result_multipliers = [_np.prod(mp) for mp in self._multipliers]
+        dr = _xr.DataArray(value_array)
 
-        result_multiplier = 1.0
-        for multiplier in self._multipliers:
-            result_multiplier = result_multiplier * multiplier
+        for (index, multiplier) in enumerate(result_multipliers):
+            if (len(self.date_range) != 0):
+                # Date is given in DD-MM, convert to MM-DD for comparison
+                start = self._convert_datestr(self.date_range[index][0])
+                end = self._convert_datestr(self.date_range[index][1])
+                dr_date = dr.time.dt.strftime(r"%m-%d")
+                dr = _xr.where((start < dr_date) & (dr_date < end), dr * multiplier, dr)
+            else:
+                dr = dr * multiplier
+        return dr
 
-        return _xr.DataArray(value_array * result_multiplier)
+    def _convert_datestr(self, date_str: str) -> str:
+        parsed_str = _dt.strptime(date_str, r"%d-%m")
+        return parsed_str.strftime(r"%m-%d")

decoimpact/business/workflow/model_builder.py

@@ -74,6 +74,7 @@ def _create_rule(rule_data: IRuleData) -> IRule:
                 [rule_data.input_variable],
                 rule_data.multipliers,
                 rule_data.output_variable,
+                rule_data.date_range
             )
 
         if isinstance(rule_data, ILayerFilterRuleData):

decoimpact/data/api/i_multiply_rule_data.py

@@ -23,5 +23,10 @@ def input_variable(self) -> str:
 
     @property
     @abstractmethod
-    def multipliers(self) -> List[float]:
+    def multipliers(self) -> List[List[float]]:
         """Name of the input variable"""
+
+    @property
+    @abstractmethod
+    def date_range(self) -> List[List[str]]:
+        """Array with date ranges"""

decoimpact/data/api/i_response_curve_rule_data.py

@@ -17,13 +17,13 @@ class IResponseCurveRuleData(IRuleData, ABC):
     @abstractmethod
     def input_variable(self) -> str:
         """Property for the input variable"""
-    
+
     @property
     @abstractmethod
     def input_values(self) -> List[float]:
         """Property for the input values"""
-    
+
     @property
     @abstractmethod
     def output_values(self) -> List[float]:
-        """Property for the output values"""
+        """Property for the output values"""

decoimpact/data/dictionary_utils.py

@@ -60,10 +60,10 @@ def convert_table_element(table: List) -> Dict:
 
     if len(headers) != len(set(headers)):
         seen = set()
-        dupes = [x for x in headers if x in seen or seen.add(x)] 
+        dupes = [x for x in headers if x in seen or seen.add(x)]
         raise ValueError(
             f"There should only be unique headers. Duplicate values: {dupes}"
         )
 
     values = list(map(list, zip(*table[1:])))  # transpose list
-    return dict(zip(headers, values))
+    return dict(zip(headers, values))

decoimpact/data/entities/multiply_rule_data.py

@@ -18,21 +18,28 @@ class MultiplyRuleData(IMultiplyRuleData, RuleData):
     def __init__(
         self,
         name: str,
-        multipliers: List[float],
+        multipliers: List[List[float]],
         input_variable: str,
         output_variable: str = "output",
-        description: str = ""
+        description: str = "",
+        date_range: List[List[str]] = [],
     ):
         super().__init__(name, output_variable, description)
         self._input_variable = input_variable
         self._multipliers = multipliers
+        self._date_range = date_range
 
     @property
     def input_variable(self) -> str:
         """Name of the input variable"""
         return self._input_variable
 
     @property
-    def multipliers(self) -> List[float]:
-        """Name of the input variable"""
+    def multipliers(self) -> List[List[float]]:
+        """List of list with the multipliers"""
         return self._multipliers
+
+    @property
+    def date_range(self) -> List[List[str]]:
+        """List of list with start and end dates"""
+        return self._date_range

decoimpact/data/parsers/parser_multiply_rule.py

@@ -8,9 +8,12 @@
 
 from decoimpact.crosscutting.i_logger import ILogger
 from decoimpact.data.api.i_rule_data import IRuleData
-from decoimpact.data.dictionary_utils import get_dict_element
+from decoimpact.data.dictionary_utils import convert_table_element, get_dict_element
 from decoimpact.data.entities.multiply_rule_data import MultiplyRuleData
 from decoimpact.data.parsers.i_parser_rule_base import IParserRuleBase
+from decoimpact.data.parsers.validation_utils import (
+    validate_all_instances_number, validate_start_before_end, validate_type_date
+)
 
 
 class ParserMultiplyRule(IParserRuleBase):
@@ -32,23 +35,29 @@ def parse_dict(self, dictionary: Dict[str, Any], logger: ILogger) -> IRuleData:
         """
         name = get_dict_element("name", dictionary)
         input_variable_name = get_dict_element("input_variable", dictionary)
-        multipliers = get_dict_element("multipliers", dictionary)
-
-        if not all(isinstance(m, (int, float)) for m in multipliers):
-            message = (
-                "Multipliers should be a list of int or floats, "
-                f"received: {multipliers}"
-            )
-            position_error = "".join(
-                [
-                    f"ERROR in position {index} is type {type(m)}. "
-                    for (index, m) in enumerate(multipliers)
-                    if not isinstance(m, (int, float))
-                ]
-            )
-            raise ValueError(f"{position_error}{message}")
         output_variable_name = get_dict_element("output_variable", dictionary)
 
+        multipliers = [get_dict_element("multipliers", dictionary, False)]
+        date_range = []
+
+        if not multipliers[0]:
+            multipliers_table = get_dict_element("multipliers_table", dictionary)
+            multipliers_dict = convert_table_element(multipliers_table)
+            multipliers = get_dict_element("multipliers", multipliers_dict)
+            start_date = get_dict_element("start_date", multipliers_dict)
+            end_date = get_dict_element("end_date", multipliers_dict)
+
+            validate_type_date(start_date, "start_date")
+            validate_type_date(end_date, "end_date")
+            validate_start_before_end(start_date, end_date)
+
+            date_range = list(zip(start_date, end_date))
+        validate_all_instances_number(sum(multipliers, []), "Multipliers")
+
         return MultiplyRuleData(
-            name, multipliers, input_variable_name, output_variable_name
+            name,
+            multipliers,
+            input_variable_name,
+            output_variable_name,
+            date_range=date_range
         )

decoimpact/data/parsers/validation_utils.py

@@ -0,0 +1,82 @@
+"""
+Module for Validation functions
+"""
+
+from typing import List
+from datetime import datetime
+
+
+def validate_all_instances_number(data: List, name: str):
+    """Check if all instances in a list are of type int or float
+
+    Args:
+        data (List): List to check
+        name (str): Name to give in the error message
+
+    Raises:
+        ValueError: Raise an error to define which value is incorrect
+    """
+    if not all(isinstance(m, (int, float)) for m in data):
+        message = (
+            f"{name} should be a list of int or floats, "
+            f"received: {data}"
+        )
+        position_error = "".join(
+            [
+                f"ERROR in position {index} is type {type(m)}. "
+                for (index, m) in enumerate(data)
+                if not isinstance(m, (int, float))
+            ]
+        )
+        raise ValueError(f"{position_error}{message}")
+
+
+def validate_type_date(data: List[str], name: str):
+    """
+    Check if all dates in list are a datestring of format: DD-MM
+
+    Args:
+        data (str): List of date strings
+        name (str): Name of data to address in error message
+
+    Raises:
+        ValueError: Raise this error to indicate which value is not
+        a date in the proper format.
+    """
+
+    for (index, m) in enumerate(data):
+        try:
+            datetime.strptime(m, r"%d-%m")
+        except TypeError:
+            message = (
+                f"{name} should be a list of strings, "
+                f"received: {data}. ERROR in position {index} is type {type(m)}."
+            )
+            raise TypeError(message)
+        except ValueError:
+            message = (
+                f"{name} should be a list of date strings with Format DD-MM, "
+                f"received: {data}. ERROR in position {index}, string: {m}."
+            )
+            raise ValueError(message)
+
+
+def validate_start_before_end(start_list: List[str], end_list: List[str]):
+    """Validate if for each row in the table the start date is before the end date.
+
+    Args:
+        start_list (List[str]): list of dates
+        end_list (List[str]): list of dates
+    """
+
+    for (index, (start, end)) in enumerate(zip(start_list, end_list)):
+        start_str = datetime.strptime(start, r"%d-%m")
+        end_str = datetime.strptime(end, r"%d-%m").replace()
+
+        if not start_str < end_str:
+            message = (
+                f"All start dates should be before the end dates. "
+                f"ERROR in position {index} where start: "
+                f"{start} and end: {end}."
+            )
+            raise ValueError(message)