RFC: Split multi.jl into multiple files. Wrap parallel into its own module.

[amitmurthy]

A first step in refactoring and making multi.jl more modular with cleaner separation of functionality. This PR splits multi.jl into smaller files and moves all multiprocessing code into a new module Base.Parallel

[amitmurthy]

Any thoughts on this? Any reasons to NOT have a Parallel submodule.

[vchuravy]

👍 from me.

[amitmurthy]

Will merge this tomorrow.

[ViralBShah]

Now that you ask, there could be a separate module called RPC (for remotecall, fetch, Futures, etc.) that can be used to build any arbitrary distributed computing system and user-facing parallel computing models in a parallel module.

Not relevant here, but do we need to put Channels in their own module too?

[amitmurthy]

@ViralBShah , I agree with you on differentiating between a higher level API (for users) and a lower level RPC API (for library writers). However, at this time I want to merge this as is and take some more time to debate the correct way of modularization Parallel and the interfaces between each of them.

Broadly, functionally I can think of

• ClusterManagement - partially modularized via ClusterManagers.jl . Needs a rework to handle elastic computation scenarios better.
• Serialization, Transport, Topology - Option to plugin other serialization schemes? Other IO infrastructure, routing via intermediate nodes, etc.
• Communication primitives - point-to-point remote calls (existing), collective MPI style communication (not yet defined)
• User facing API - pmap, distributed arrays, etc

Not relevant here, but do we need to put Channels in their own module too?

IMO, Channels and all task related code should be grouped together under module Tasks

[vtjnash]

Any reasons to NOT have a Parallel submodule

In order to make this a toplevel module? 😛

But seriously, would it be difficult to make this a toplevel module? I think using Parallel would be nicer than using Base.Parallel, and would better align with making this a stdlib module.

[vtjnash]

Can delete this list (* means done)

[vchuravy]

@vtjnash That is the long term-goal, but should we start moving bits and pieces out before we have the infrastructure for stdlib?

[vtjnash]

We should be able to make them toplevel modules without yet changing anything about the stdlib infrastructures. I expect this may be one of the harder ones to detangle, however. The PR here will be a useful start to that process.

[amitmurthy]

As a first step, I'll just rebase and merge as is. Will take a shot at making it a top-level module in a different PR.

The main issues in distentangling that I see are:

• current behavior of using that loads on all workers. using and import only load on current process #19073 may remove the need for this.
• loading from node 1
• workers preloading the Parallel module when they come up

[amitmurthy]

I am going to rework this again in a few days to make it mergeable. Would like to get it in in the 0.6 timeframe. No user facing functionality change. Just code reorg.

[tkelman]

still planning on doing this?

[amitmurthy]

Yes. I am waiting for other PRs to get in / not get in before redoing this PR.

[StefanKarpinski]

You may want to refresh this change so it can be merged since it's not externally visible and then rebase the parallel for change on top of this.

[StefanKarpinski]

Not externally visible – can be done whenever.

[amitmurthy]

Closing in favor of #20428

As mentioned there, we should get this in soon. Will help in backporting bugfixes onto 0.6.