Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drop AppPipeline in favor of pipeline namespaces #8

Closed
takikadiri opened this issue Jan 20, 2024 · 0 comments · Fixed by #14
Closed

Drop AppPipeline in favor of pipeline namespaces #8

takikadiri opened this issue Jan 20, 2024 · 0 comments · Fixed by #14
Assignees

Comments

@takikadiri
Copy link
Owner

takikadiri commented Jan 20, 2024

kedro-boot introduces AppPipeline which is a kedro Pipeline that have some additional metadata used by kedro-boot as a specs for the compilation process.

Kedro's pipeline terminorlogy can already be confusing. Adding another Kind of Pipeline will make kedro-boot hard to grasp for users, and make it harder for us to maintain and document. What if we can infer the kedro-boot compilation specs only from the Pipeline object, leveraging namespaces. Here is a potential mapping between AppPipeline attributes and the Pipeline infered attributes:

AppPipeline name --> Pipeline namespace or 'None' if no namespace
AppPipeline inputs --> namespaced free inputs datasets
AppPipeline outputs --> namespaced outputs datasets
AppPipeline parameters --> namespaced parameters datasets
AppPipeline artifacts --> Automatically infered with this spec : free inputs - namespaced free inputs (artifacts inference can be disabled)

This will allow us to leverage also kedro-viz, as we could visualize theses attributes :

namespace_example

Here we see clearly which datasets that will be "touched" by the kedro boot apps (the namespaced part, the blue one). The "regressor" in this case is not a namespaced free inputs datasets, it will be infered as an artifacts.

Note that kedro boot apps could explicitely give their own compilation specs, which take precedence over these automatically infered specs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant