Add --conf-source option to kedro ipython

[ankatiyar]

Description

Right now we can pass env and params as options with kedro ipython / %reload_kedro. Add --conf-source option as well.

Context

User question from slack :

Speaking of kedro ipython :
Is there a way to specify /override the conf_source ? 🙂

My problem comes from the fact that we have a single pipeline / repo for all our clients.
To better enforce “tenant isolation”
I have organized conf and data as follow:
conf
└── client_A
├── base
└── local
└── client_B
├── base
└── local
data
├── client_A
│ └── 01_raw
│ └── 02_intermediate
│ └── ...
└── client_B
│ └── 01_raw
│ └── 02_intermediate
│ └── ...conf
I did so because I liked the idea of not being able to run the pipeline without being explicit about which client it should be done for…
(the structure above makes it so that kedro run without --conf-source will raise an error)
Another reason I did so (if I recall correctly) was to avoid having “duplicate keys” conflicts in the different yaml files across clients…

Possible Implementation

Changes to be made in kedro/ipython/__init__.py
PR for when --conf-source was added to kedro run - #2117

[astrojuanlu]

xref https://linen-slack.kedro.org/t/13906421/hi-again-speaking-of-kedro-ipython-is-there-a-way-to-specify#4b179fdf-0b5c-48cb-b287-6a53ed5b3bbb

[laizaparizotto]

Hi @ankatiyar, I would need some help to tackle this issue.

I read the PR #2117, but I am unsure how this should be implemented for ipython.

Is it that it should be first added as an @argument("--conf-source", type=str, default=None, help=CONF_SOURCE_HELP) here?

kedro/kedro/ipython/__init__.py

Line 62 in e1823ed

@argument(

Then in function def reload_kedro add conf-source: str = None as an input

kedro/kedro/ipython/__init__.py

Line 81 in e1823ed

extra_params: dict[str, Any] = None,

And should be used as an input to load kedroSession

session = KedroSession.create(
        metadata.package_name, project_path, env=env, extra_params=extra_params, conf-source=conf-source
    ) 

kedro/kedro/ipython/__init__.py

Line 93 in e1823ed

session = KedroSession.create(

Please let me know what your ideas are.. 💡

[ankatiyar]

@laizaparizotto I think what you've suggested looks about right, might be some more things are required. I haven't looked into the implementation myself but I can and in the meantime, if you can try this out too and open a PR(or a draft PR), we could collaborate on the implementation.

[laizaparizotto]

Sure, @ankatiyar.

I will make the changes and open a PR soon. Thank you for the help