Feature/routine for decomissioning of exisiting capacity #78

FelixMau · 2024-03-15T12:28:54Z

Adding in Decommissioning for stock processes fading out.

Detect processes (via "0" name end)
Check on existing max values
Unify capacity values to one largest capacity occouring

…missioning-of-exisitng-capacity

& results from industy test

henhuy · 2024-04-15T10:28:02Z

data_adapter_oemof/calculations.py

+    if max_column in adapter_dict.keys():
+        if adapter_dict[capacity_column] == adapter_dict[max_column]:
+            adapter_dict[max_column] = adapter_dict[capacity_column] / np.max(
+                adapter_dict[capacity_column]
+            )
+        else:
+            logging.info("Decommissioning and max value can not be set in parallel")
+            adapter_dict[max_column] = list(
+                (adapter_dict[max_column] / np.max(adapter_dict[capacity_column]))
+            )
+    else:
+        adapter_dict[max_column] = adapter_dict[capacity_column] / np.max(
+            adapter_dict[capacity_column]
+        )


Can you explain whats happening in decommission function, as I don't quite get it from code alone?
Best to put in docstring directly...

Baisicly I am just dividing the yealy changing capacity by the maximum capacity and then setting the max capacity as value for all years while using the quotient as value for max

I also wanted to check whether max value already is used and wheter it just is the same as the capacity value (as in data provided for steel industy)

FelixMau · 2024-04-15T14:21:01Z

Separating caluclaitons to happen pre and post calculation.

Recalculation of activity bonds to be relative values. And then applying decommission logic helps to be certain about already existing max values.

Moreover making it easier to adapt in future if more unexpected values occour.

…g-capacity # Conflicts: # data_adapter_oemof/adapters.py

data_adapter_oemof/adapters.py

henhuy

Hey Felix, thank you for starting to implement this.
I made some comments/requests for you.
One thing: the max/min in MIMO will look different and will not depend on output_parameters - just to keep that in mind.

data_adapter_oemof/calculations.py

data_adapter_oemof/adapters.py

FelixMau · 2024-04-17T10:55:34Z

Made following changes:

Rearranged Functions for better overview & understanding
Implemented pre mapping calculations (to normalize activity bonds and map fix to min and max)
Revisited Decomissioning function and adapting to now known input if max value was already existing

FelixMau · 2024-04-17T11:14:06Z

Will be adapting tests on dev to keep things at least a little separated.
Going to add empty dictionaries for input and output parameters

henhuy

Implementation looks good.
I have only smaller things regarding user feedback via logging, which could help us debugging.

data_adapter_oemof/calculations.py

FelixMau · 2024-06-10T09:41:59Z

Now including a nan value handler to replace nan values automatically if the parameter values differ over the periods.

Replacing max values by 9999999999999.
Replacing min and all other values by 0.

This feature is not meant to be used extensively since leaving values empty for some years is implicit and in most cases considered 0!

henhuy

Have two questions

henhuy · 2024-06-18T10:33:57Z

data_adapter_oemof/calculations.py

+        if column in ["method", "source", "comment", "bandwidth_type"]:
+            continue
+
+        if group_df[column].nunique(dropna=False) > 1:


What does this line?

It should make sure to only apply cahnges to columns with nan values. I modified to make the intention more clear.

Somehow, GitHub marked the wrong line. I meant the part if group_df[column].nunique(dropna=False) > 1: ?

henhuy · 2024-06-18T10:34:26Z

data_adapter_oemof/calculations.py

+        if group_df[column].nunique(dropna=False) > 1:
+            if column in max:
+                group_df[column].fillna(max_value, inplace=True)
+            elif column in min:


This could be removed, as next else statement looks the same

Modified to raise an error if nan values are found in columns that cannot be derived as min/max.

Multi-Period cannot handle nan values for some Periods only as far as I know.

henhuy · 2024-06-18T12:38:04Z

data_adapter_oemof/calculations.py

@@ -214,10 +214,16 @@ def handle_nans(group_df: pd.DataFrame) -> pd.DataFrame:
        if column in ["method", "source", "comment", "bandwidth_type"]:
            continue

-        if group_df[column].nunique(dropna=False) > 1:
+        if group_df.isna().sum>0:


I would remove this check completely as I think this costs more than simply fillna all columns which need to be checked...

henhuy · 2024-06-18T12:40:08Z

data_adapter_oemof/calculations.py

+                if "type" in group_df.columns:
+
+                    raise ValueError(f"In column {column} for process {group_df['type']} nan values are found"
+                                     f"please make sure to only provide complete datasets")
+                else:
+                    raise ValueError(f"In column {column} nan values are found"
+                                     f"please make sure to only provide complete datasets")


This could cause us more trouble then letting None in there IMO.
So maybe not check for it instead...

Yes it would but yet I am facing this issue again now with x2x dataset and it is hard to guess which dataset is faulty. The dataset has empty lists where also non empty lists are.
This data set has nans and lists in the same column.

Having an Error here or in data_adapter would help since it is actually an error? We should not ignore and detect it somewhere (as early as possible)?

But maybe this should be discussed in another issue?

Having an Error here or in data_adapter would help since it is actually an error? We should not ignore and detect it somewhere (as early as possible)?

But I think there will be many cases in datasets where this is the case - and I don't see people fixing datasets quick enough. So better be prepared for None values and ignore them as far as possible (tabular sets default values in most cases)?

Tabular will not be able to set default values since currently it is impossible to know which values are missing. We are loosing information if single entries are missing because tabular is receiving a list for changing values. If the list is shorter than the amount of periods we will not know anymore which years value has gone missing.

Maybe we can adjust the reading of datapackage in tabular to use pass a nan if a string is (eg. "nan") is found in such a list?

FelixMau · 2024-06-24T12:10:04Z

Due to unrelated changes returning to #78 d9e872d

Continueing to work on handling nans on #89

FelixMau added 8 commits March 5, 2024 14:53

Structure for steelcast test case

47a629e

Merge Dev

192cb71

start test writing

f51db2d

Adapt strukture and Mappings for steel

b832178

Merge branch 'feature/add_mimo_adapter' into feature/routine-for-deco…

4da45c7

…missioning-of-exisitng-capacity

Add decomissioning to adapters & calculations

4cc72fd

Linting

32cf342

Warn and return single capacity

82c57e0

FelixMau self-assigned this Mar 15, 2024

nailend changed the title ~~Feature/routine for decomissioning of exisitng capacity~~ Feature/routine for decomissioning of exisiting capacity Mar 27, 2024

nailend mentioned this pull request Mar 27, 2024

Implement decommissioning via max value oemof/oemof-tabular#166

Open

nailend assigned SabineHaas Mar 27, 2024

FelixMau added 4 commits April 3, 2024 13:14

Calc max value from capacity

89ac703

Commenting & Linting

4e28f88

Adding empty helper

97a7ad0

& results from industy test

Commenting & Linting

39192f5

FelixMau requested review from henhuy and SabineHaas April 10, 2024 13:53

FelixMau added 2 commits April 11, 2024 10:54

Loggin info if max already set

c8b1f35

Loggin info if max already set

cf958cb

Base automatically changed from feature/add_mimo_adapter to dev April 15, 2024 10:22

henhuy reviewed Apr 15, 2024

View reviewed changes

Adding comment

9583a68

FelixMau marked this pull request as draft April 15, 2024 14:16

FelixMau linked an issue Apr 15, 2024 that may be closed by this pull request

Routine for Decomissioning of exisitng capacity #65

Closed

FelixMau added 4 commits April 15, 2024 16:22

Seperating calculation functionality

4cbeb22

Moreover making it easier to adapt in future if more unexpected values occour.

seperating functions for better overview

0ee4eca

Merge branch 'dev' into feature/routine-for-decomissioning-of-existin…

1e3d412

…g-capacity # Conflicts: # data_adapter_oemof/adapters.py

Move post mappinig calculations to calculations

6132c83

FelixMau added 2 commits April 16, 2024 12:03

Commenting & Linting

8c0c690

Idea pitch to add in flow parameters

4cee6ff

FelixMau commented Apr 16, 2024

View reviewed changes

data_adapter_oemof/adapters.py Outdated Show resolved Hide resolved

henhuy requested changes Apr 16, 2024

View reviewed changes

FelixMau added 2 commits April 17, 2024 12:14

Rearranging functions

e32c109

Revisting decomissioning

d96d6f6

Test fail due to empty parameter dicts

2901bb9

FelixMau marked this pull request as ready for review April 17, 2024 11:12

FelixMau added 4 commits April 24, 2024 10:31

Update oemof.industry

3451937

Update oemof.industry

78600e5

Adjust extra fields for mimo

03fbefc

Remove outputparameters in Mimo

60c135f

FelixMau requested a review from henhuy May 6, 2024 09:07

Update dependencies

acdb9e9

henhuy requested changes May 15, 2024

View reviewed changes

data_adapter_oemof/calculations.py Outdated Show resolved Hide resolved

data_adapter_oemof/calculations.py Outdated Show resolved Hide resolved

data_adapter_oemof/calculations.py Outdated Show resolved Hide resolved

data_adapter_oemof/calculations.py Outdated Show resolved Hide resolved

Update documentation & Logging

d9e872d

FelixMau requested a review from henhuy May 23, 2024 07:43

henhuy reviewed Jun 18, 2024

View reviewed changes

henhuy requested changes Jun 18, 2024

View reviewed changes

FelixMau force-pushed the feature/routine-for-decomissioning-of-exisitng-capacity branch from 92635d0 to d9e872d Compare June 24, 2024 12:05

FelixMau mentioned this pull request Jun 24, 2024

Feature/handle nans #89

Merged

henhuy approved these changes Jun 24, 2024

View reviewed changes

FelixMau merged commit ffa4845 into dev Jun 25, 2024
4 checks passed

FelixMau deleted the feature/routine-for-decomissioning-of-exisitng-capacity branch June 25, 2024 07:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/routine for decomissioning of exisiting capacity #78

Feature/routine for decomissioning of exisiting capacity #78

FelixMau commented Mar 15, 2024 •

edited

Loading

henhuy Apr 15, 2024

FelixMau Apr 15, 2024

FelixMau commented Apr 15, 2024

henhuy left a comment

FelixMau commented Apr 17, 2024

FelixMau commented Apr 17, 2024

henhuy left a comment

FelixMau commented Jun 10, 2024

henhuy left a comment

henhuy Jun 18, 2024

FelixMau Jun 18, 2024

henhuy Jun 18, 2024

henhuy Jun 18, 2024

FelixMau Jun 18, 2024

henhuy Jun 18, 2024

henhuy Jun 18, 2024

FelixMau Jun 19, 2024 •

edited

Loading

henhuy Jun 19, 2024

FelixMau Jun 20, 2024 •

edited

Loading

FelixMau commented Jun 24, 2024 •

edited

Loading

Feature/routine for decomissioning of exisiting capacity #78

Feature/routine for decomissioning of exisiting capacity #78

Conversation

FelixMau commented Mar 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FelixMau commented Apr 15, 2024

henhuy left a comment

Choose a reason for hiding this comment

FelixMau commented Apr 17, 2024

FelixMau commented Apr 17, 2024

henhuy left a comment

Choose a reason for hiding this comment

FelixMau commented Jun 10, 2024

henhuy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FelixMau Jun 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FelixMau Jun 20, 2024 • edited Loading

Choose a reason for hiding this comment

FelixMau commented Jun 24, 2024 • edited Loading

FelixMau commented Mar 15, 2024 •

edited

Loading

FelixMau Jun 19, 2024 •

edited

Loading

FelixMau Jun 20, 2024 •

edited

Loading

FelixMau commented Jun 24, 2024 •

edited

Loading