using and import only load on current process #19073
Conversation
|
Since this is breaking it should have a line in |
amitmurthy
force-pushed
the
amitm/using
branch
from
06f3a92 to
57d6f9a
Compare
|
Updated. Would like a review by @vtjnash before merging. |
| @@ -221,7 +220,8 @@ Module file paths | |||
| ----------------- | |||
|
|
|||
| The global variable LOAD_PATH contains the directories Julia searches for | |||
| modules when calling ``require``. It can be extended using ``push!``:: | |||
| modules when using ``using`` (or ``import``) is invoked. | |||
| @@ -170,15 +170,19 @@ the following code:: | |||
| Starting Julia with ``julia -p 2``, you can use this to verify the following: | |||
|
|
|||
| - :func:`include("DummyModule.jl") <include>` loads the file on just a single process (whichever one executes the statement). | |||
| - ``using DummyModule`` causes the module to be loaded on all processes; however, the module is brought into scope only on the one executing the statement. | |||
| - ``@everywhere import DummyModule`` causes the module to be loaded on all processes. However, it is not brought into scope on | |||
There was a problem hiding this comment.
is the module not brought into scope, or is this referring to the exports of the module (using vs import distinction) ?
There was a problem hiding this comment.
Only the exports. Good catch.
|
Also, currently |
|
Bump @vtjnash |
| else | ||
| content = remotecall_fetch(open, node, read, path_to_try) | ||
| restored = _include_from_serialized(path_to_try) |
There was a problem hiding this comment.
Can remove the toplevel_load bool and just always use this codepath. The other path is only relevant if it needed to also broadcast the file to all workers.
| else | ||
| eval(Main, :(Base.include_from_node1($path))) | ||
| end | ||
| eval(Main, :(Base.include_from_node1($path))) |
There was a problem hiding this comment.
This risks making the node module state inconsistent since the behavior of __precompile__ is not uniform across nodes, resulting in a race condition that this block of code previously tried to heuristically avoid. To solve, I think we need to remove the myid test from __precompile__, but also to make it possible to ignore a __precompile__ statement (perhaps some global dictionary of sourcepath names for which __precompile__ is temporarily disabled?). Then when we catch a precompilableerror, we can first request the cache compile file from node 1 (the package_locks synchronization might be able to help synchronize these requests). Once we know if that will succeed for fail, we can proceed with either loading the cache file, or include it with __precompile__ disabled for that file. (this should also be changed to be the case on node 1, not just worker nodes, as I think this logic should be more robust – ignoring the cache whenever it is unusable – but just hadn't been implemented yet)
|
would be great to have this in 0.6. |
|
This was incomplete. Closed it. |
|
#22588 didn't address the |
As discussed in JuliaLang/Distributed.jl#20, this PR changes the behavior of
usingto only load on the current process.cc: @vtjnash