Installable Kedro catalog (Mini-Kedro)

[WaylonWalker]

Would it make sense to make mini-kedro installable? My use case for projects like that are users doing EDA and just want easy access to the data with no fuss.

If it is something that makes sense I propose adding a setup.py to make it installable, and a single module that sets the catalog up for them, then they can access the project's data as follows.

import my_mini_kedro as mmk
mmk.catalog.load('my_dataset')

Alternatively

This could also be a separate starter that is a kedro-catalog starter.

[JavierHernandezMontes]

@WaylonWalker do you know a way to load a certain dataset of the catalog with the current version of kedro 0.17.0? as you propose to do.

[WaylonWalker]

I just added a very basic setup.py and an __init__.py with a catalog loader to make it work.

[WaylonWalker]

@JavierHernandezMontes is your data local or remote? For local data that needs packaged in it might be a bit trickier.

[noklam]

@WaylonWalker Is this still relevant with the IPython extension?

[astrojuanlu]

At the moment, using Kedro without using the project template is entirely possible: pip install kedro and then instantiating the catalog directly:

from kedro.config import OmegaConfigLoader
from kedro.io import DataCatalog

conf_loader = OmegaConfigLoader("conf")
conf_catalog = conf_loader.get("catalog")
catalog = DataCatalog.from_config(conf_catalog)

catalog.load(...)

On IPython & Jupyter, this is one line:

%load_ext kedro.ipython

catalog.load(...)

I agree it would be nice to make the boilerplate above go away, so I'm moving this issue and renaming it for our consideration. It will have more visibility on the framework repo.

[astrojuanlu]

More evidence of users using the Kedro catalog without the pipelines: #2898 (comment)

[astrojuanlu]

Another user asking for exactly this https://linen-slack.kedro.org/t/16593946/is-there-a-way-of-installing-only-the-data-catalog-part-of-k#b6d532c4-2d7f-4add-b0ee-b0bfcffbdd5e

[astrojuanlu]

Interesting realisation today. Users have been able to use the DataCatalog as a standalone component since basically forever. The mini-kedro starter was created in Kedro 0.17.0:

https://github.com/kedro-org/kedro-starters/blob/0.17.0/mini-kedro/%7B%7B%20cookiecutter.repo_name%20%7D%7D/Example%20Notebook.ipynb

Then in 2023 we made a big push to show that you can use Kedro standalone components (#2855, #3128)

And yet, even some of our power users had no idea it's possible to do this.

In #3659 I gave a detailed rationale of the technical reasons why it's just better to have fewer dependencies and enable users to install only the parts they want. But I was missing the marketing reasons: old users have preconceived notions about what Kedro can and cannot do, and new users find a wall of documentation explaining the framework way, so most don't realise that there's a library way as well.

[astrojuanlu]

This is still important for us.

As part of a issue cleanup, and in line with a new scheme in which we want to use Discussions for feature requests & enhancement proposals #3767, I'm moving this to a discussion. Let's continue there.