Can we remove ParallelRunner?

[merelcht]

Description

Check telemetry to find out how much ParallelRunner is used and whether it would be possible to deprecate and remove it.

Context

ParellelRunner is the most problematic among our runners:

• kedro-viz produces warnings when run project with ParallelRunner #4078
• Viz hook is broken with ParallelRunner [Blocked by Framework]  kedro-viz#1801
• Rethink how Kedro can play a role in multiprocessing / performance boost #3713
• ParalleRunner hangs on Linux Server #4176
• [Investigate] ParallelRunner does not work with S3-linked data catalog #2162
• ParallelRunner raises AttributeError: The following data sets cannot be used by multiprocessing... on datasets not involved in --pipeline being run #3804
• Investigate why Spaceflights project failing with ParallelRunner  #3674
• [KED-3075] Using CachedDataSet and ParallelRunner together fails #673

And there's already a bit of evidence it's not used that much: https://linen-slack.kedro.org/t/16663577/do-you-use-kedro-run-runner-parallerunner-to-speed-up-your-p#99abccb0-7970-4a65-8fad-85fd22681beb

[astrojuanlu]

We discussed that there is indeed evidence that some people at least try to use this runner, but because it's so broken, they're rarely successful or they have to battle workarounds.

We agreed to do some research to understand the underlying issues and see if the ParallelRunner is still the best solution to them.

[yury-fedotov]

I'm using ParallelRunner in most projects... Surprised if its unpopular, based on telemetry

[astrojuanlu]

Thanks @yury-fedotov, that's good to know. Does it work well for your needs? Have you been hit by any issues?

[tejaassolanki]

Even I have used it in my previous organisation and it is most likely still being used. If I remember correctly, the telemetry data won't show up from that organisation.

[astrojuanlu]

Unsure if this is being recorded in the telemetry at all. The prompt for this issue was a Slack thread that isn't necessarily representative.

I'm mostly ready to say that we should fix the issues instead of deprecating the runner.

[yury-fedotov]

Thanks @yury-fedotov, that's good to know. Does it work well for your needs? Have you been hit by any issues?

Yes, works well, and I haven't encountered issues