Fix workflow for running Africa

[davide-f]

Closes # (if applicable).

Changes proposed in this Pull Request

I openned this PR to describe current efforts in enabling the workflow to run for Africa.
This PR integrates #284

Checklist

• I tested my contribution locally and it seems to work fine.
• Code and workflow changes are sufficiently documented.
• Newly introduced dependencies are added to envs/environment.yaml and envs/environment.docs.yaml.
• Changes in configuration options are added in all of config.default.yaml, config.tutorial.yaml, and test/config.test1.yaml.
• Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/configuration.rst and doc/tutorial.rst.
• A note for the release notes doc/release_notes.rst is amended in the format of previous release notes, including reference to the requested PR.

[davide-f]

@doneachh @Eddy-JV @hazemakhalek @energyLS this PR is ready for a first revision of the procedure and highlights.
But do not merge yet

[davide-f]

FYI this PR has run for Africa locally with the custom data stored in the parallel PR opened here, but the model is infeasible.
Numbers need checking

[hazemakhalek]

The PR looks alright to me, I am not able to run it though as I'm currently out of office. All the logic seem alright to me so if you tested it @davide-f and the CI passes I think we can merge it

[davide-f]

The PR looks alright to me, I am not able to run it though as I'm currently out of office. All the logic seem alright to me so if you tested it @davide-f and the CI passes I think we can merge it

Great to hear! and many thanks :)

I'll wait merging it till at least this is merged.
That leads to some little issues in the execution that can apply to -sec.
I'm locally testing it for Africa (-sec version) and World (only power).

[davide-f]

This has worked for Africa, but with mathematical unfeasibility though; Ready to review and finalize :)

[davide-f]

Many thanks! @doneachh !
One last update to be merged after review in pypsa-earth though:
pypsa-meets-earth/pypsa-earth#992

[davide-f]

Moreover, this PR needs some checks: for MA it doesn't work well. I'm investigating

[davide-f]

@davide-f could you summarize for me, which tasks are open at this point for getting serious africa modeling results?

• model doesnt have enough biomass potential
• Missing H2-Buses for GHA
• Something else as well?

Hello :D
Sorry my late reply, I didn't see your review but I've answered all your comments.

Regarding the rest:

• I've rerun the model with the latest changes, including both increased biomass (4e6 TWh/a), biogass (2e5 TWh/a) and oil (1e5 TWh) potentials --> the problem still is infeasible (or it has numerical issues)
• I confirm some missing buses for some gas pipelines that need some fixing. I'd recommend that we could improve the modelling in pypsa-earth so that when using the gadm clustering, each region shall have at least a bus. This should fix this problem

Overall, I feel like there is the need for some debugging for larger regions.
I'd recommend to start "easy": rather then modelling Africa as a whole, maybe start with a power pool, then follow to other power pools and finally the whole model. That is to get confidence of the results at regional level.
We need to check how the model performs with multiple countries: in this PR I aimed to fix the limitations I saw in prepare_sector_network that was coded to represent a single country, but there may be other limitations in the whole code. That needs some checking

[davide-f]

For me this PR is ready to go.
I'd recommend somebody to rerun it locally as well on MA to have a crosscheck.

[davide-f]

I think I discovered the issue and the problem relates to isolated nodes.
What I did is several testing. The latest one that helped me achieve this insight is:

1. run MA only with 20 nodes -> working
2. run MA+DZ with 20 nodes -> failing due to numerical issues
3. run MA+DZ with 15 nodes and varying CO2 limits -> failing
4. run DZ only with 10 nodes -> failing
5. checked the DZ-only network that was supposed to have only 10 nodes and instead it has 36

I see 2 options:

1. merge this PR
2. first rerun a MA scenario again comparing original and new approach and merge it

[doneachh]

I think I discovered the issue and the problem relates to isolated nodes. What I did is several testing. The latest one that helped me achieve this insight is:

1. run MA only with 20 nodes -> working
2. run MA+DZ with 20 nodes -> failing due to numerical issues
3. run MA+DZ with 15 nodes and varying CO2 limits -> failing
4. run DZ only with 10 nodes -> failing
5. checked the DZ-only network that was supposed to have only 10 nodes and instead it has 36

I see 2 options:

1. merge this PR
2. first rerun a MA scenario again comparing original and new approach and merge it

@davide-f Great to hear that you have found the issue! :) Do you already know why DZ has 36 nodes instead of the 10 specified in the config?

What do you mean with original and new approach?

From my side we can merge this PR! :)

[davide-f]

I'd propose to merge the PR as is and then revise it later; otherwise the workflow is deviating too much.

A last review is needed to check merge conflicts

[doneachh]

@davide-f Code looks good to me! You can merge it, if you want :)

[davide-f]

Thanks @doneachh !

@hazemakhalek following the discussion.
To verify no additional issue is added here, this PR has been further tested for MA using upstream main and this PR.
The objective value changes slightly, about 1-2%, the reasons are the following:

1. this PR fixes a bug in the heat demand that was previously omitted
2. the update of the submodule has changes slightly something

If that's fine for you, we can merge. I'm rerunning the PR to check if it still works for Africa given the latest changes but it contains most improvements needed. Minor issues may be left

[doneachh]

@davide-f Fine for me!

[davide-f]

Africa has successfully executed locally :) merging
Thanks Anton! :D