Issue with powerplantsmatching

[ekatef]

I'm currently setting the model from scratch and have some misunderstandings with powerplantsmatching. Would be very grateful for any suggestions.

Checklist

• I am using the current main branch or the latest release. Please indicate.
• I am running on an up-to-date pypsa-africa environment. Update via conda env update -f envs/environment.yaml.

Describe the Bug

Run of the build_powerplants rule leads to an AssertionError: No matches found caused by pm.powerplants(from_url=False, update=True, config=config). All settings in the config correspond to the config.default.yaml except countries: ["CD"] and retrieve_databundle: false.

The data files with loaded from zenodo manuallly (there were some troubles during automated loading probably due to some tricks of network connection). But structure of the data directories corresponds to one implied by the model and it doesn't look suspicious...

Error message

If applicable, paste any terminal output to help illustrating your problem.
In some cases it may also be useful to share your list of installed packages: conda list.

INFO:snakemake.logging:
INFO:pypsa.io:Imported network base.nc has buses, lines
INFO:powerplantmatching.collection:Create combined dataset for GEO, GPD
INFO:powerplantmatching.matching:Comparing data sources `GEO` and `GPD`
Traceback (most recent call last):
  File "~/Documents/_github_/pypsa-africa/.snakemake/scripts/tmpo7x4285k.build_powerplants.py", line 251, in <module>
    pm.powerplants(from_url=False, update=True, config=config)
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.10/site-packages/powerplantmatching/collection.py", line 223, in matched_data
    matched = collect(matching_sources, config=config, **collection_kwargs)
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.10/site-packages/powerplantmatching/collection.py", line 97, in collect
    matched = combine_multiple_datasets(dfs, datasets, config=config, **dukeargs)
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.10/site-packages/powerplantmatching/matching.py", line 258, in combine_multiple_datasets
    crossmatches = link_multiple_datasets(datasets, labels, config=config, **dukeargs)
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.10/site-packages/powerplantmatching/matching.py", line 208, in link_multiple_datasets
    return cross_matches(all_matches, labels=labels)
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.10/site-packages/powerplantmatching/matching.py", line 147, in cross_matches
    assert not matches.empty, "No matches found"
AssertionError: No matches found
[Sat May 28 14:33:48 2022]
INFO:snakemake.logging:[Sat May 28 14:33:48 2022]
Error in rule build_powerplants:
    jobid: 0
    output: resources/powerplants.csv, resources/powerplants_osm2pm.csv
    log: logs/build_powerplants.log (check log file(s) for error message)

ERROR:snakemake.logging:Error in rule build_powerplants:
    jobid: 0
    output: resources/powerplants.csv, resources/powerplants_osm2pm.csv
    log: logs/build_powerplants.log (check log file(s) for error message)

RuleException:
CalledProcessError in line 329 of ~/Documents/_github_/pypsa-africa/Snakefile:
Command 'set -euo pipefail;  ~/opt/miniconda3/envs/pypsa-africa/bin/python3.10 ~/Documents/_github_/pypsa-africa/.snakemake/scripts/tmpo7x4285k.build_powerplants.py' returned non-zero exit status 1.
  File "~/Documents/_github_/pypsa-africa/Snakefile", line 329, in __rule_build_powerplants
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.10/concurrent/futures/thread.py", line 58, in run
ERROR:snakemake.logging:RuleException:
CalledProcessError in line 329 of ~/Documents/_github_/pypsa-africa/Snakefile:
Command 'set -euo pipefail;  ~/opt/miniconda3/envs/pypsa-africa/bin/python3.10 ~/Documents/_github_/pypsa-africa/.snakemake/scripts/tmpo7x4285k.build_powerplants.py' returned non-zero exit status 1.
  File "~/Documents/_github_/pypsa-africa/Snakefile", line 329, in __rule_build_powerplants
  File "~/opt/miniconda3/envs/pypsa-africa/lib/python3.10/concurrent/futures/thread.py", line 58, in run
Removing output files of failed job build_powerplants since they might be corrupted:
resources/powerplants_osm2pm.csv
WARNING:snakemake.logging:Removing output files of failed job build_powerplants since they might be corrupted:
resources/powerplants_osm2pm.csv
Shutting down, this might take some time.
WARNING:snakemake.logging:Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
ERROR:snakemake.logging:Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2022-05-28T143341.241785.snakemake.log
WARNING:snakemake.logging:Complete log: .snakemake/log/2022-05-28T143341.241785.snakemake.log

[davide-f]

Thanks @ekatef for reporting the issue!
Recently powerplantmatching has been updated, maybe it is a problem related to the version you are using.
You may try to reinstall powerplantmatching and, if that doesnt work, you may try to reinstall the environment.
To Uninstaller powerplantmatching, you may:

pip uninstall powerplantmatching
pip install git+https://github.com/davide-f/powerplantmatching.git@new_pypsa_africa#egg=powerplantmatching

[ekatef]

@davide-f, many thanks for the suggestion and instructions! :)

[ekatef]

UPD: Have tried both reinstallation of powerplantmatching and recreation of the environment from scratch. The error is not fixed yet but it seems that the problem does in fact relates by the way powerplantmatching is working on my system.

During the build_powerplants.py run, the generators are successfully extracted from raw osm data into csv_pm and seem to be properly loaded to the disk using the filepath_osm2pm_ppl path. The config is also being updated with config["EXTERNAL_DATABASE"]["fn"]. But after that a call of the powerplantmatching's powerplants( ) function(which is essentially matched_data( )) by some mystical reason results in an empty entry and throws an error. It looks like a powerplantmatching can't find the custom files.

By the way, @davide-f could please you verify please if the following line is applicable globally:
https://github.com/pypsa-meets-africa/pypsa-africa/blob/69b077c2179500d3ece34d314079a2c75e86c121/configs/powerplantmatching_config.yaml#L29
Or should it be adjusted locally? Although, after some experimenting it looks like the problem is rather in some system variables used by powerplantmatching :)

UPD2: The glpk solver is also not particularly happy by the system variables with the FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/qn/vpndfm21795ckkq89np1ckp40000gn/T/pypsa-solvehyvzupo8.sol'.

It looks like some system-dependent additional adjustments may be needed to make the whole workflow run if energy modelling packages are not pre-installed.

[davide-f]

This error is surprising.
With the new update, the EXTERNAL_DATABASE is not included in the workflow, hence the standard ppl processing is executed.
Not sure if it may be a problem specific to CD (I try to reproduce it as well).

Regarding the gplk is the first time we experience such issue; is that problem reproducible? have you experienced that problem also before?

[ekatef]

@davide-f, many thanks for looking into this.

No, the issue with gplk is not familiar for me.

Regarding the problem specifics you are absolutely right: it seems that only one generator remains in the dataframe after clean_osm_data is applied. I'll test if the problem persists on different regions. Thank you for the suggestion!

[davide-f]

I can reproduce the issue as well; I think it is a bug in ppl to be investigated.
For the time being, I'd recommend to debug adding an additional country or changing the countries to avoid having empty matches, in the meantime that the issue is fixed

[ekatef]

That's great to know that the issue is reproducible :) @davide-f, thank you so much for your help!

[davide-f]

I've investigated the issue.
The problem is a matter of naming conventions in powerplantmatching (ppl).
For DRC in GEO database, ppl is expected to provide the name "Democratic Republic of Congo", whereas, our workflow provides him with "Congo, Democratic Republic of". This leads to empty databases and missing data that need to be fixed.
I'll investigate it.

[ekatef]

@davide-f That's genial :) I'ld say not obvious at all...

[ekatef]

I'm facing this issue again after returning to the mainstream powerplantmatching. The environment is freshly built using the latest environment specification.

@davide-f, would be very grateful for any ideas on how to fix it :)

[davide-f]

Install the version from main, not the conda release.
Maybe that's the reason

[ekatef]

@davide-f, thanks for the suggestion! The issue was resolved with changing the source of ppm installation from conda distribution to installation from ppm repo via pip.

In case this issue is reproducible, does it probably makes sense to update the environment settings?

[pz-max]

@ekatef , ppm get soon an update. Let's wait

[ekatef]

@pz-max, great to know! Thanks for the information. Actually, it looks like the updated ppm version works like a charm :)

Have checked Vietnam where a similar issue was experienced previously, and it appears that the current main-ppm version successfully resolves that as well.

Just in case the temporary fix is to replace the source for powerplantmatching in environment.yaml from - powerplantmatching>=0.5.5 to
`- pip:

• git+https://github.com/PyPSA/powerplantmatching@master`