Merge pull request #41 from JuliaDebug/teh/world_age

Fix world age problems when running tests

Manifest.toml

@@ -3,9 +3,9 @@
 [[Base64]]
 uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"
 
-[[Distributed]]
-deps = ["Random", "Serialization", "Sockets"]
-uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b"
+[[Dates]]
+deps = ["Printf"]
+uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"
 
 [[InteractiveUtils]]
 deps = ["Markdown"]
@@ -15,12 +15,9 @@ uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240"
 deps = ["Base64"]
 uuid = "d6f4376e-aef5-505a-96c1-9c027394607a"
 
-[[Random]]
-deps = ["Serialization"]
-uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
+[[Printf]]
+deps = ["Unicode"]
+uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"
 
-[[Serialization]]
-uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"
-
-[[Sockets]]
-uuid = "6462fe0b-24de-5631-8697-dd941f90decc"
+[[Unicode]]
+uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"

Project.toml

@@ -6,9 +6,10 @@ version = "0.1.1"
 InteractiveUtils = "b77e0a4c-d291-57a0-90e8-8db25a27a240"
 
 [extras]
+Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
 Distributed = "8ba89e20-285c-5b6f-9357-94700520ee1b"
 Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
 Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
 
 [targets]
-test = ["Test", "Distributed", "Random"]
+test = ["Test", "Distributed", "Random", "Dates"]

docs/Project.toml

@@ -2,3 +2,4 @@
 Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
 InteractiveUtils = "b77e0a4c-d291-57a0-90e8-8db25a27a240"
 JuliaInterpreter = "aa1ae85d-cabe-5617-a682-6adf51b2e16a"
+Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

docs/make.jl

@@ -1,4 +1,4 @@
-using Documenter, JuliaInterpreter
+using Documenter, JuliaInterpreter, Test
 
 makedocs(
     modules = [JuliaInterpreter],

docs/src/dev_reference.md

@@ -15,6 +15,8 @@ JuliaInterpreter.build_frame
 JuliaInterpreter.determine_method_for_expr
 JuliaInterpreter.prepare_args
 JuliaInterpreter.prepare_call
+JuliaInterpreter.prepare_thunk
+JuliaInterpreter.prepare_toplevel
 JuliaInterpreter.get_call_framecode
 JuliaInterpreter.optimize!
 ```
@@ -26,7 +28,9 @@ JuliaInterpreter.Compiled
 JuliaInterpreter.step_expr!
 JuliaInterpreter.finish!
 JuliaInterpreter.finish_and_return!
+JuliaInterpreter.get_return
 JuliaInterpreter.next_until!
+JuliaInterpreter.through_methoddef_or_done!
 JuliaInterpreter.evaluate_call!
 JuliaInterpreter.evaluate_foreigncall!
 JuliaInterpreter.maybe_evaluate_builtin
@@ -45,10 +49,12 @@ JuliaInterpreter.JuliaProgramCounter
 ```@docs
 JuliaInterpreter.framedict
 JuliaInterpreter.genframedict
+JuliaInterpreter.compiled_methods
 ```
 
 ## Utilities
 
 ```@docs
+JuliaInterpreter.@lookup
 JuliaInterpreter.iswrappercall
 ```

docs/src/internals.md

@@ -1,5 +1,7 @@
 # Internals
 
+## Basic usage
+
 The process of executing code in the interpreter is to prepare a `frame` and then
 evaluate these statements one-by-one, branching via the `goto` statements as appropriate.
 Using the `summer` example described in [Lowered representation](@ref),
@@ -150,4 +152,117 @@ julia> frame.ssavalues
 ```
 
 One can easily continue this until execution completes, which is indicated when `step_expr!`
-returns `nothing`.
+returns `nothing`. Alternatively, use the higher-level `JuliaInterpreter.finish!(stack, frame)`
+to step through the entire frame,
+or `JuliaInterpreter.finish_and_return!(stack, frame)` to also obtain the return value.
+
+## More complex expressions
+
+Sometimes you might have a whole sequence of expressions you want to run.
+In such cases, your first thought should be `prepare_thunk`.
+Here's a demonstration:
+
+```jldoctest; setup=(using JuliaInterpreter; empty!(JuliaInterpreter.junk))
+using Test
+
+ex = quote
+    x, y = 1, 2
+    @test x + y == 3
+end
+
+frame = JuliaInterpreter.prepare_thunk(Main, ex)
+JuliaInterpreter.finish_and_return!(JuliaStackFrame[], frame)
+
+# output
+
+Test Passed
+```
+
+## Toplevel code and world age
+
+Code that defines new `struct`s, new methods, or new modules is a bit more complicated
+and requires special handling. In such cases, calling `finish_and_return!` on a frame that
+defines these new objects and then calls them can trigger a
+[world age error](https://docs.julialang.org/en/latest/manual/methods/#Redefining-Methods-1),
+in which the method is considered to be too new to be run by the currently compiled code.
+While one can resolve this by using `Base.invokelatest`, we'd have to use that strategy
+throughout the entire package.  This would cause a major reduction in performance.
+To resolve this issue without leading to performance problems, care is required to
+return to "top level" after defining such objects. This leads to altered syntax for executing
+such expressions.
+
+Here's a demonstration of the problem:
+
+```julia
+ex = :(map(x->x^2, [1, 2, 3]))
+frame = JuliaInterpreter.prepare_thunk(Main, ex)
+julia> JuliaInterpreter.finish_and_return!(JuliaStackFrame[], frame)
+ERROR: this frame needs to be run a top level
+```
+
+The reason for this error becomes clearer if we examine `frame` or look directly at the lowered code:
+
+```julia
+julia> Meta.lower(Main, ex)
+:($(Expr(:thunk, CodeInfo(
+1 ─      $(Expr(:thunk, CodeInfo(
+1 ─     global ##17#18
+│       const ##17#18
+│       $(Expr(:struct_type, Symbol("##17#18"), :((Core.svec)()), :((Core.svec)()), :(Core.Function), :((Core.svec)()), false, 0))
+└──     return
+)))
+│   %2 = (Core.svec)(##17#18, Core.Any)
+│   %3 = (Core.svec)()
+│   %4 = (Core.svec)(%2, %3)
+│        $(Expr(:method, false, :(%4), CodeInfo(quote
+    (Core.apply_type)(Base.Val, 2)
+    (%1)()
+    (Base.literal_pow)(^, x, %2)
+    return %3
+end)))
+│        #17 = %new(##17#18)
+│   %7 = #17
+│   %8 = (Base.vect)(1, 2, 3)
+│   %9 = map(%7, %8)
+└──      return %9
+))))
+```
+
+All of the code before the `%7` line is devoted to defining the anonymous function `x->x^2`:
+it creates a new "anonymous type" (here written as `##17#18`), and then defines a "call
+function" for this type, equivalent to `(##17#18)(x) = x^2`.
+
+In some cases one can fix this simply by indicating that we want to run this frame at top level:
+
+```julia
+julia> JuliaInterpreter.finish_and_return!(JuliaStackFrame[], frame, true)
+3-element Array{Int64,1}:
+ 1
+ 4
+ 9
+```
+
+Here's a more fine-grained look at what's happening under the hood (and a robust strategy
+for more complex situations where there may be nested calls of new methods):
+
+```julia
+modexs, _ = JuliaInterpreter.prepare_toplevel(Main, ex)
+stack = JuliaStackFrame[]
+for (mod, e) in modexs
+    frame = JuliaInterpreter.prepare_thunk(mod, e)
+    while true
+        JuliaInterpreter.through_methoddef_or_done!(stack, frame) === nothing && break
+    end
+    JuliaInterpreter.get_return(frame)
+end
+```
+
+This splits the expression into a sequence of frames (here just one, but more complex blocks may be split up into many).
+Then, each frame is executed until it finishes defining a new method, then returns to top level.
+The return to top level causes an update in the world age.
+If the frame hasn't been finished yet (if the return value wasn't `nothing`),
+this continues executing where it left off.
+
+(Incidentally, `JuliaInterpreter.enter_call(map, x->x^2, [1, 2, 3])` works fine on its own,
+because the anonymous function is defined by the caller---you'll see that the created frame
+is very simple.)