Enable/Disable hook with runtime config

[foxale]

Description

It may be useful to turn off certain hooks for specific kedro runs.

Context

Let's say I have a hook, that when kedro run is invoked, automatically profiles all outputs for me and stores HTML reports somewhere in the project dir. Of course, this is time consuming and I don't want to do have this feature enabled on each run but on demand. What's the best way to turn in on and off?

Possible Implementation / Alternatives

Should I use --params and implement some switch on the hook side? Or maybe we should have a way to just decide which hooks to activate per run?

Inspired by #435

[noklam]

You can disable installed hooks via settings.py via

# settings.py
# DISABLE_HOOKS_FOR_PLUGINS = ("kedro-viz",)`

Or are you specifically requesting this for the Kedro CLI?

[foxale]

I would want to adjust project source code only if that's a change that should be version controlled down the line (on project level), and leave all single-run-related config to Kedro CLI and yaml run configs.

[noklam]

I assume this is similar to [Allow extra hooks to be passed via the Kedro CLI, update the title slightly.

Quick question:
Since we follow a pytest opt-out model, plugin hooks are installed automatically. Do you need to disable a hook at runtime? This may work by using some kind of environment variable in the hook implementation?

[foxale]

Correct. The best combo though, would be a CLI parameter + env variable. In the ideal world that would work like so:

kedro run --with-profiling

where --with-profiling would be passed to my custom ProfilingHook() class indicating user preference.

[noklam]

Added some relevant notes from Discord today.

ToniMaroni — Today at 8:57 AM
Hi, is there a way to have the configurations in settings.py configurable based on the kedro environment. For example, I want the hooks for kedro-viz and kedro-neptune disabled in production environment. Thanks
datajoely — Today at 8:59 AM
Interesting usecase - best solution off the top of my head is to use the KEDRO_ENV environment variable and just include an if statement in settings.py
ToniMaroni — Today at 9:00 AM
I was also thinking of a using an ENV var, glad to hear that there is one already defined for the KEDRO_ENV! Thanks
datajoely — Today at 9:01 AM
The other option, which is probably preferable, is to do the logic in the hooks themselves which do have the env name as a parameter
This isn't possible if you're using 3rd party hooks

[noklam]

Correct. The best combo though, would be a CLI parameter + env variable. In the ideal world that would work like so:

kedro run --with-profiling

where --with-profiling would be passed to my custom ProfilingHook() class indicating user preference.

@foxale I think supporting arbitrary CLI arguments in the official Kedro CLI is unlikely.

To achieve that, you can actually override Kedro CLI in your project easily. The cli.py is actually part of Kedro Project Template in the older version - but we remove it since most people don't need to customize it.

I think you can achieve this by doing the following

1. Override the CLI with cli.py , add your new CLI arguments
2. Inject the argument as an environment variable
3. In your hooks/plugin - it should read this environment variable and behave accordingly.

See an example here:
https://github.com/noklam/kedro_gallery/blob/master/kedro_cli_extension/src/kedro_cli_extension/cli.py

[foxale]

Awesome. And my assumption is you can implement such changes to CLI via kedro plugins as well?

[merelcht]

@foxale You could also point DISABLE_HOOKS_FOR_PLUGINS in settings.py to read from an environment variable.

[merelcht]

I'm closing this issue since there hasn't been any recent activity. Feel free to re-open this if you're still facing problems!