Initialise PySpark using hooks rather than custom context

[antonymilne]

Spun out of #506 (comment).

Currently PySpark is initialised in kedro using a custom context. We now have a much better place to do this: after_context_created hook defined in hooks.py. This would look something like this:

class SparkHooks:
   @hook_impl
   def after_context_created(self, context) -> None:
        """Initialises a SparkSession using the config
        defined in project's conf folder.
        """

        # Load the spark configuration in spark.yaml using the config loader
        parameters = context.config_loader.get("spark*", "spark*/**")
        spark_conf = SparkConf().setAll(parameters.items())

        # Initialise the spark session
        spark_session_conf = (
            SparkSession.builder.appName(self._package_name)
            .enableHiveSupport()
            .config(conf=spark_conf)
        )
        _spark_session = spark_session_conf.getOrCreate()
        _spark_session.sparkContext.setLogLevel("WARN")

In addition to making the hooks.py file you should remove the context.py file and edit settings.py to instantiate SparkHooks in HOOKS and no longer provide a custom context.

We need to change this in a:

1. Both pyspark and pandas-pyspark starters
2. Documentation

Maybe in future SparkHooks will live somewhere within the kedro package so that users can just do from kedro.extras.hooks.pyspark import SparkHooks and not need to define the hook themselves.

Related/possible alternative? #904

[antonymilne]

@Galileo-Galilei comment:

Lim's comment ("There used to be 2 kinds of hooks: registration & life-cycle. Managing them using the same hook managers was a mistake") and the difficulties you have to locate this new hook is in line with #904 and the "engines" design patterns: registering objects is not the same as calling them during session lifecycle. kedro.extras is likely the right place for now, but it likely need some more thoughts on the design later.

[datajoely]

I've been thinking it may be nice to have some sort of AbstractExecutionContext that we could use for Spark, Dask, Snowpark and others ...

[antonymilne]

@datajoely could those also be put into an after_context_created hook? I'm not sure what the code for those would look like, so don't know exactly which kedro objects would be required to instantiate them.

e.g. all the spark instantiation requires access to is context.config_loader, and even that I'm not sure we really need (does anyone actually want to change spark.yml for different environments? Would be good to know).

[datajoely]

I think you have to assume you would want local/dev/prod spark configuration environments. I think all of these can be migrated to this pattern, my push here is to make sure we think about remote execution targets in abstract when developing this.

Personally - I'm very keen to build a Snowpark implementation, but will only do so once this stabilises.

[merelcht]

Completed in kedro-org/kedro-starters#102