--scenarios Parameter Causes Broken DAGs

[khelfen]

Description

Using the CLI parameter --scenarios results in broken DAGs due to incorrect conversion of the parameter into egon-data.configuration.yaml. This causes the following error:

Error Message

Broken DAG: [.../eGon-data/SH1/code/src/egon/data/airflow/dags/pipeline.py]
Traceback (most recent call last):
  File "<string>", line 7, in __init__
  File ".../code/src/egon/data/datasets/__init__.py", line 277, in __post_init__
    last = list(self.tasks.last)[0]
IndexError: list index out of range

Cause

The parameter --scenarios is being incorrectly written into the YAML file as:

--scenarios: '[''status2019'', ''eGon2035'']'

Expected Behavior

The parameter should be correctly translated into YAML as:

--scenarios:
- status2019
- eGon2035

Steps to Reproduce

1. Use the CLI to run a scenario.
2. Observe the resulting YAML and notice the incorrect format.
3. Try to run the pipeline and observe the broken DAG error.

Impact

• The pipelines are unable to execute due to invalid configurations.
• Manual correction of the YAML file is required as a workaround.

Proposed Solution

Adjust the conversion logic to ensure the --scenarios parameter is written into the YAML file in the correct format.

[ClaraBuettner]

I have not started a new run for a couple of months, but I remember that we once had similar problems.
@CarlosEpia: Do you know if that was fixed?

[CarlosEpia]

I think we faced that problem at some point. Since we always work with powerd-data by creating the egon-data.configuration.yaml in advance, the required changes were probably never done.