prepare_sector_network.py: pandas.errors.IntCastingNaNError: Cannot convert non-finite values (NA or inf) to integer

[fhg-isi]

Describe the Bug

I tried to run

python scripts/prepare_sector_network.py

and got the error below

Error Message

(pypsa-eur) projekt-resilient03@ubuntu-22-04-lts-temp:~/pypsa-eur$ python scripts/prepare_sector_network.py
Output_
/home/projekt-resilient03/pypsa-eur/results/prenetworks/elec_s_37_lv1.0__730H-T-H-B-I-A-dist1_2050.nc
INFO:pypsa.io:Imported network elec_s_37_ec_lv1.0_.nc has buses, carriers, generators, global_constraints, lines, links, loads, shapes, storage_units, stores
INFO:__main__:Removing buses with carrier ['H2', 'battery']
INFO:__main__:Removing links with carrier ['H2 electrolysis', 'H2 fuel cell', 'battery charger', 'battery discharger']
INFO:__main__:Removing generators with carrier ['CCGT', 'OCGT', 'biomass', 'coal', 'geothermal', 'lignite', 'oil']
INFO:__main__:Removing stores with carrier ['H2', 'battery']
INFO:__main__:Added connection cost of 8834-39750 Eur/MW/a to offwind-dc
INFO:__main__:Added connection cost of 3950-25835 Eur/MW/a to offwind-ac
INFO:__main__:Added connection cost of 5182-85173 Eur/MW/a to offwind-float
WARNING:pypsa.components:Bus has no attribute location, ignoring this passed value.
WARNING:pypsa.components:Bus has no attribute location, ignoring this passed value.
INFO:__main__:Adding electricity generation
INFO:__main__:Add hydrogen storage
INFO:__main__:Adding hydrogen fuel cell for re-electrification.
INFO:__main__:Add hydrogen underground storage
INFO:__main__:Add options for new hydrogen pipelines.
INFO:__main__:Add land transport
INFO:__main__:fuel_cell share: 0%
INFO:__main__:electric share: 100%
INFO:__main__:ice share: 0%
INFO:__main__:Add heat sector
INFO:__main__:Assumed space heat reduction of 29.00%
INFO:__main__:Add biomass
INFO:__main__:Add industrial demand
INFO:__main__:Add possibility to use industrial waste heat in district heating
INFO:__main__:Add agriculture, forestry and fishing sector.
ERROR:root:Uncaught exception
Traceback (most recent call last):
  File "/home/projekt-resilient03/pypsa-eur/scripts/prepare_sector_network.py", line 4057, in <module>
    n = set_temporal_aggregation(
        ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/projekt-resilient03/pypsa-eur/scripts/prepare_sector_network.py", line 3662, in set_temporal_aggregation
    .astype(int)
     ^^^^^^^^^^^
  File "/home/projekt-resilient03/conda/envs/pypsa-eur/lib/python3.11/site-packages/pandas/core/generic.py", line 6643, in astype
    new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/projekt-resilient03/conda/envs/pypsa-eur/lib/python3.11/site-packages/pandas/core/internals/managers.py", line 430, in astype
    return self.apply(
           ^^^^^^^^^^^
  File "/home/projekt-resilient03/conda/envs/pypsa-eur/lib/python3.11/site-packages/pandas/core/internals/managers.py", line 363, in apply
    applied = getattr(b, f)(**kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/projekt-resilient03/conda/envs/pypsa-eur/lib/python3.11/site-packages/pandas/core/internals/blocks.py", line 758, in astype
    new_values = astype_array_safe(values, dtype, copy=copy, errors=errors)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/projekt-resilient03/conda/envs/pypsa-eur/lib/python3.11/site-packages/pandas/core/dtypes/astype.py", line 237, in astype_array_safe
    new_values = astype_array(values, dtype, copy=copy)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/projekt-resilient03/conda/envs/pypsa-eur/lib/python3.11/site-packages/pandas/core/dtypes/astype.py", line 182, in astype_array
    values = _astype_nansafe(values, dtype, copy=copy)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/projekt-resilient03/conda/envs/pypsa-eur/lib/python3.11/site-packages/pandas/core/dtypes/astype.py", line 101, in _astype_nansafe
    return _astype_float_to_int_nansafe(arr, dtype, copy)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/projekt-resilient03/conda/envs/pypsa-eur/lib/python3.11/site-packages/pandas/core/dtypes/astype.py", line 145, in _astype_float_to_int_nansafe
    raise IntCastingNaNError(
pandas.errors.IntCastingNaNError: Cannot convert non-finite values (NA or inf) to integer

Related:
#1118

[ulfmueller]

I run into the same error once I use the nH trigger in the {sector_opts} wildcard. Is there any meaningful workaround or hint?

[fhg-isi]

Might be related to the geopandas issue. (Did not try; just occured at the same time). => Ensure that you are using latest version or try geopandas==0.14.4.
Also see #1136

[ulfmueller]

Thanks for mentioning! Unfortunately that didn't do the trick for me...

[ulfmueller]

Some additional information: I looked a bit into the code. If I set e.g. 3h or 3H in the sector_opts, in the set_temporal_aggregation() it finds that there is no 'sn' in the resolution (which is instead 3h) and then takes a csv file for the snapshot_weightings which in my case is an empty file which then leads to nan values causing the problem here.

I tried out a few quick and dirty workarounds. One, entering a '3sn' to the sector_opts config seems to work... But I do not know if it leads to any other unwanted changes. E.g. I was wondering if this aggregation part would be missing in this case.

[cristobal-GC]

You can try to set the temporal resolution here:

pypsa-eur/config/config.default.yaml

Line 858 in d2e7562

resolution_sector: false

replacing 'false' with 'nH'.