Expose internal structure of the Runner?

[astrojuanlu]

As discussed, I will be moving this ticket to backlog, as we cannot access the SyncManager instance from the hooks to register a shared dict with the manager that is started with ParallelRunner. So, we need some way of exposing the manager (either through the catalog or runner in Kedro) and make it mutable for the custom hooks.

from @ravi-kumar-pilla kedro-org/kedro-viz#1801 (comment)

cc @noklam

On a related not, I sense that something is off with the ParallelRunner, seeing lots of issues about it lately but I couldn't keep up and collect them.

Related:

• https://github.com/kedro-org/kedro/milestone/28

[noklam]

One thing that we should expose is RUNNER_ARGS in settings.py (although custom runner is not that common). I'd also focus more on Runner in general but not ParallelRunner. I express this in #3713 earlier.

[noklam]

The No.1 issue is coming from the "fork" vs "spawn" process, in fact nothing has changed and it has been always broken. The only thing change is that Before Python 3.8, Mac's default is "fork" process so everything runs fine locally and in CI. Since then things start breaking in MacOS too.

See: #3705

[noklam]

On ParallelRunner, I was working with @ravi-kumar-pilla on the ticket. The problem is to make something works in multiprocess requires the use of SyncManger.

In Mac (>3.8) and Windows, modules get re-imported because they are "spawned" but not "fork". As a result hooks in settings.py is recreated in the middle of Kedro run. There are no way that hook have access to the SyncManager, which is currently stored inside the Runner as runner._manager.

We have more detailed discussion which is not included in #1801. historically, Runner creates the default dataset, thus the SyncManager stays with the Runner object. Since late 0.19, runner no longer create default datasets, it get moved to DataCatalog with a default dataset pattern. For ParallelRunner we never move this away and the real default dataset definition resides on ParallelRunner.

[noklam]

One thing that we can do is to try to move this back to DataCatalog. However, this doesn't resolve the issue that Hook have no access of the SyncManager.