register_pipelines hook specifications in kedro 0.18 ?

[ed-rhilbert]

First time posting ! Congrats and thanks to the Kedro team and community for this tool 😃

Has the register_pipelines hook specification been removed in kedro 0.18 ? I was using it to automatically build pipelines in a plugin in 0.17.7, is there any workaround in 0.18 ?

[datajoely]

Hello if you check out the release notes for 0.18.0 we explain the reasoning and new way it works:

https://github.com/kedro-org/kedro/blob/main/RELEASE.md

[ed-rhilbert]

Hello,

Thanks for the quick answer ! I checked and found explanations for register_config_loader and register_catalog, but not for register_pipelines ... or did not understand ? Any example to migrate this specific hook would be greatly appreciated.

[antonymilne]

Hi @ed-rhilbert, this is a great question and you're the first person to ask it! register_pipelines as a hook specification has indeed been removed (the work to do this was in progress even before 0.18.0, since the register_pipelines function in pipeline_registry.py was no longer a hook implementation but instead just a regular Python function).

You're actually the first person I've heard of who was using register_pipelines in a plugin. Personally I quite liked this functionality also, but the decision was made that it was not the right way to share pipelines. The new way to do this is micro-packaging, e.g. kedro micropkg pull path/to/pipeline-0.1.tar.gz -d pipelines.

Some of the advantages of this are:

• it's more general than just packaging pipelines (e.g. can be used for utilities)
• developing and distributing pipelines is much easier than making a plugin since you don't need to understand how to make a whole Python package, you just need to use kedro micropkg package
• it includes aliasing if you want to change the name of the pipeline/package
• it includes some (currently quite basic) handling of a requirements.txt file if you have different dependencies for different pipelines
• you can package and pull in bulk using pyproject.toml and the --all flag

I'd be interested in hearing if this functionality can be easily used to replace your plugin's register_pipelines hook or if there's something you're not able to do any more. In general, if the old flow you were using was:

• pip install kedro
• kedro install
• pip install my-pipelines-plugin
• kedro run --pipeline=blah

The new flow would be:

• pip install kedro
• pip install -r src/requirements.txt (since kedro install no longer exists)
• kedro micropkg pull
• Manually modify register_pipelines in pipeline_registry.py to register newly pulled pipelines
• kedro run --pipeline=blah (note you can also now use -p instead of --pipeline)

Two further things to note:

• if you are were using a plugin to register multiple pipelines then I really recommend modifying the pyproject.toml file and then importing in bulk using the --all flag. You can even make a custom starter template which includes a customised pyproject.toml file listing pipelines if you always pull the same ones in your projects
• I'm hoping to make this flow smoother by automatically registering pipelines so you don't need to do the step "Manually modify register_pipelines in pipeline_registry.py". I'm also wondering whether we should add something to settings.py to make customised pipeline registration easier. Please do see Design auto-registration of pipelines #1284 and leave comments if you have thoughts on this. As I said, you're the only person I've heard of who used the old functionality so I'd be very interested in hearing whether the new solution would work well for you or if we're missing something!

[antonymilne]

@ed-rhilbert: this makes a lot of sense, thanks for the explanation. As @datajoely said, you aren't the first person to try and come up with yaml-defined pipelines (here at QuantumBlack we have some big projects that like to do this). I'm guessing that other people who have done this have not been using the register_pipelines hook or they also would have complained that we've removed it.

Do you like to define your pipeline.yml on a per-environment basis (like conf/base/pipeline.yml) or do you have just one definition (src/project_name/pipeline.yml)?

Either way, since register_pipelines didn't have any particular arguments available to it as a hook, you should be able to achieve the same behaviour as your plugin just through modifying the register_pipelines function in pipeline_registry.py. Let's say you pip install kedro_yaml_pipelines. Then you would do something like:

import kedro_yaml_pipelines

def register_pipelines() -> Dict[str, Pipeline]:
    return kedro_yaml_pipelines.make_pipelines("pipeline.yml")

As far a I can tell, you should be able to do the same as you currently do - the only difference is that you will need to manually edit the pipeline_registry.py file, which I know is a bit annoying. If this is a change you want to make to a lot of projects you might like to make your own starter template that has your custom pipeline_registry.py file in it.

[ed-rhilbert]

Hello to both of you,

@datajoely , your answer makes total sense. I would not be an advocate for a yml definition of the pipelines in Kedro, but it can prove useful for our cases and users here at Eurodecision. For my information, what does DSL mean ?

@AntonyMilneQB , your proposal is exactly what I have done and I confirm it is ok. A bit disapointing compared to the plugins/hooks mechanism that I find really great, but ok ... We will go through with this.

Thanks and, imho, pipeline auto-registration would be a great feature !

[antonymilne]

Awesome, thanks for the feedback @ed-rhilbert. And sorry that this solution is not as neat as it used to be using the register_pipeline hook - I do appreciate how annoying that must be 😬 I think part of the reason these hooks were removed was that it was deemed too easy/insecure to inject custom behaviour just by pip installing something and then not requiring any manual intervention to activate it.

Let's try and go down the route of pipeline auto-registration and exposing the register_pipeline function in settings.py then. This way you would just have something like this:

# settings.py
REGISTER_PIPELINE_FUNCTION = kedro_yaml_pipelines.make_pipelines

... which still requires a manual edit to a file after pip installing your plugin, but I think is more consistent with the way you can inject other custom components (like a custom config loader).

[IanVlasov]

Hello to all of you!
I have a pretty similar use case to what ed-rhilbert introduced, and I would like to give a comment/question:
Do you think it is good idea to use environment variables to pass "pipeline.yml" to kedro_yaml_pipelines.make_pipelines function?

e.g. I would have path:path/to/pipeline.yml inside my globals.yml and by using custom ConfigLoader or maybe by invoking before_command_run CLI hook I will add this path to the environment variable e.g. KEDRO_PIPELINE_PATH and then pass this variable to the kedro_yaml_pipelines.make_pipelines:

kedro run --params path:path/to/pipeline.yml

import os
import kedro_yaml_pipelines

def register_pipelines() -> Dict[str, Pipeline]:
    return kedro_yaml_pipelines.make_pipelines(
        os.getenv(KEDRO_PIPELINE_PATH)
    )

[antonymilne]

Hello @IanVlasov, as you probably realise unfortunately there's not going to be a very satisfactory way to do this because register_pipelines doesn't have direct access to any arguments that are passed to kedro run. So however you do it you'll need to somehow get the information magically injected into register_pipelines in a somewhat magical/ugly way.

My suggested way would be to make an after_context_created hook that makes global pipeline_path set to context.params["path"]. This path can then be set either in a parameters.yml file in the appropriate environment or overridden with kedro run --params pipeline_path:path/to/pipeline.yml. Then import the global pipeline_path in your register_pipelines function

This is quite similar to what you suggested, only it uses a Python global variable rather than OS environment variable, which feels a little bit cleaner. I think the important thing is that extraction of the path from params is probably best done in after_context_created, where you already have access to params, rather than before_command_run, where you would need to parse CLI arguments yourself to extract it.