Harmonisation

[znichollscr]

First cut of harmonisation

[znichollscr]

@jkikstra while you're reviewing, can you double check the changes to notebooks/0202_IAMC_IAMregionAggregations.py. I had to make them to make everything line up.I think they're all fine and make sense, but I wasn't 100% sure.

[jkikstra]

In my review, I did:

• run all notebooks: all 010*, 0201, 0202
• download scenarios from the ssp-submission database following the notebook
• visually check all code
• added the gcb data and download instructions
• added the edgar data and download instructions
• some minor fixes (e.g. SF6 from EDGAR, not from WMO as unavailable?)

these are in

I did not manage to finish:

• 0300; this gets an error when trying to load gcages.harmonisation.Harmoniser

C:\Users\kikstra\AppData\Local\Temp\ipykernel_24176\3185700512.py:10: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  import tqdm.autonotebook as tqdman

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[1], line 11
      9 import seaborn as sns
     10 import tqdm.autonotebook as tqdman
---> 11 from gcages.harmonisation import Harmoniser
     12 from gcages.pre_processing import PreProcessor
     13 from gcages.units_helpers import strip_pint_incompatible_characters_from_units

File [~\Documents\GitHub\emissions_harmonization_historical\gcages\src\gcages\harmonisation\__init__.py:15](http://localhost:8889/lab/tree/notebooks/~/Documents/GitHub/emissions_harmonization_historical/gcages/src/gcages/harmonisation/__init__.py#line=14)
     13 from gcages.aneris_helpers import harmonise_all
     14 from gcages.harmonisation.helpers import add_harmonisation_year_if_needed
---> 15 from gcages.parallelisation import (
     16     assert_only_working_on_variable_unit_variations,
     17     run_parallel,
     18 )
     21 def harmonise_scenario(
     22     indf: pd.DataFrame,
     23     history: pd.DataFrame,
   (...)
     26     calc_scaling_year: int,
     27 ) -> pd.DataFrame:
     28     """
     29     Harmonise a scenario
     30 
   (...)
     58         Harmonised `indf`
     59     """

File [~\Documents\GitHub\emissions_harmonization_historical\gcages\src\gcages\parallelisation.py:53](http://localhost:8889/lab/tree/notebooks/~/Documents/GitHub/emissions_harmonization_historical/gcages/src/gcages/parallelisation.py#line=52)
     44     if variations_in_other_cols.shape[0] > 1:
     45         raise AssertionError(f"{variations_in_other_cols=}")
     48 def run_parallel(
     49     func_to_call: Callable[Concatenate[U, P], T],
     50     iterable_input: Iterable[U],
     51     input_desc: str,
     52     n_processes: int,
---> 53     mp_context: BaseContext | None = multiprocessing.get_context("fork"),
     54     *args: P.args,
     55     **kwargs: P.kwargs,
     56 ) -> tuple[T, ...]:
     57     """
     58     Run a function in parallel
     59 
   (...)
    105         in combination with `args` and `kwargs`.
    106     """
    107     if n_processes == 1:

File [~\Documents\GitHub\emissions_harmonization_historical\.pixi\envs\default\Lib\multiprocessing\context.py:243](http://localhost:8889/lab/tree/notebooks/~/Documents/GitHub/emissions_harmonization_historical/.pixi/envs/default/Lib/multiprocessing/context.py#line=242), in DefaultContext.get_context(self, method)
    241     return self._actual_context
    242 else:
--> 243     return super().get_context(method)

File [~\Documents\GitHub\emissions_harmonization_historical\.pixi\envs\default\Lib\multiprocessing\context.py:193](http://localhost:8889/lab/tree/notebooks/~/Documents/GitHub/emissions_harmonization_historical/.pixi/envs/default/Lib/multiprocessing/context.py#line=192), in BaseContext.get_context(self, method)
    191     ctx = _concrete_contexts[method]
    192 except KeyError:
--> 193     raise ValueError('cannot find context for %r' % method) from None
    194 ctx._check_available()
    195 return ctx

ValueError: cannot find context for 'fork'

• 0400; because this builds on 0300

[jkikstra]

@jkikstra while you're reviewing, can you double check the changes to notebooks/0202_IAMC_IAMregionAggregations.py. I had to make them to make everything line up.I think they're all fine and make sense, but I wasn't 100% sure.

I only changed back the order to have "region" in the 3rd position, instead of the 4th, which you had changed before - but wasn't sure why.

[znichollscr]

I only changed back the order to have "region" in the 3rd position, instead of the 4th, which you had changed before - but wasn't sure why

Nice. I was probably trying to work out what was going on and forgot to change it back.

[znichollscr]

• 0300; this gets an error when trying to load gcages.harmonisation.Harmoniser

Ah yes windows doesn't have forking, I will tweak.

[znichollscr]

Thanks for the review @jkikstra, looks really good. I think we should now be able to do these merges and we're good to go: #51, #50, #43 (i.e. this one)

[jkikstra]

Now all is running through, with all the download READMEs there!
Will merge.