Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validate the . dot notation in Kedro pipeline #3080

Open
noklam opened this issue Sep 25, 2023 · 3 comments
Open

Validate the . dot notation in Kedro pipeline #3080

noklam opened this issue Sep 25, 2023 · 3 comments
Labels
Issue: Feature Request New feature or improvement to existing feature

Comments

@noklam
Copy link
Contributor

noklam commented Sep 25, 2023

Description

kedro-org/kedro-viz#1522

User are confused with disconnected graph in kedro-viz
https://docs.kedro.org/en/stable/tutorial/add_another_pipeline.html

Context

Internally . is reserved for namespace, but Kedro Framework never block this/validate this eagerly. So user can create something that looks like a namespace pipeline but isn't.

User should use the namespace argument instead of using inputs=namespace.my_input.

def _prefix_dataset(name: str) -> str:
return f"{namespace}.{name}"

Internally, dot is used for namespace. Noted that even Node have _namespace attribute but it is actually more about which pipeline it is belong to. It's ambiguous what's the namespace of a node because the input and output can come from different namespace.

Possible Implementation

Validate this behavior and block it. We probably cannot validate the node because it is a valid internal representation. We can however validate the argument in node constructor if it is desired.

Possible Alternatives

Validate this behavior and raise warning.

@noklam noklam added the Issue: Feature Request New feature or improvement to existing feature label Sep 25, 2023
@datajoely
Copy link
Contributor

There is a wider point about how we internally store namespaces in the catalog with __ and it's weird!

@MehdiNV
Copy link
Contributor

MehdiNV commented Sep 25, 2023

Leaving notes for the future:

Some validation is completed, but like Nok notes above, may not be usable as we prefix. Regardless, may be of interest in the future - check the highlighted parts below for how Kedro already attempts some validation:

image

@astrojuanlu
Copy link
Member

For now, let's add a WARNING to the logs when the user uses a . in an input or output specifier.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Issue: Feature Request New feature or improvement to existing feature
Projects
Status: No status
Development

No branches or pull requests

4 participants