Passing extra_params to override parameters specified in globals.yml in Kedro 0.18.1

[flaneur-ml]

Description

In 0.17.X it was possible to override the parameters specified in globals.yml with extra_params from cli at runtime and access it in pipeline_registry.py to configure the behaviour of the pipelines. However, when migrated to 0.18.1 it seems impossible to do due to the complete deprecation of the register_config_loader hook. I have implemented a custom config loader as suggested in #1527. However, I still struggle to access the overridden parameter in my pipeline_registry.py.

Context

In my pipeline_registry.py I want to have access to the extra_params: mode=mode1 like here:

from .settings import HOOKS

    def register_pipelines() -> Dict[str, Pipeline]:
        mode = HOOKS[0].extra_params["mode"] # select hook and extract the parameter
    
        data_engineering_pipeline = data_eng.create_pipeline(
            model_names=["m1", "m2", "m3"], mode=mode
        )
        feature_engineering_pipeline = feat_eng.create_pipeline(mode=mode)
        feature_selection_pipeline = feat_sel.create_pipeline(mode=mode)
    
        return {
            "__default__": data_engineering_pipeline
            + feature_engineering_pipeline
            + feature_selection_pipeline
        }

In 0.17.7 I was able to access mode directly from the register_config_loader hook:

class ProjectHooks:
    @hook_impl
    def register_config_loader(
        self,
        conf_paths: Iterable[str],
        env: str,
        extra_params: Dict[str, Any],
    ) -> TemplatedConfigLoader:
        self.extra_params = extra_params
        return TemplatedConfigLoader(conf_paths=conf_paths, globals_pattern="*globals.yml", globals_dict=extra_params)

Close to issues #1411, #666, #1527

Environment

• kedro version: 0.18.1
• python: 3.8.12
• Operating system: macOS Monterey 12.4

[antonymilne]

Hello @youth-quaker. This is something that is unfortunately not as easy as it used to be, but I think there's an ok way of doing it - see #1436 (reply in thread). In your case you would set a global variable mode in the after_context_created hook, and then import it into register_pipelines.

[flaneur-ml]

@AntonyMilneQB thank you! It worked out.