Incremental lowering API #84
Conversation
This is a step toward an iteration interface for lowering which can return a sequence of CodeInfo to be evaluated for top level and module expressions. This also restricts lowering of module expressions to be syntactically at top level (ie, not inside a top level thunk), consistent with the existing way that they're handled in eval.
|
I think this would be very useful API to have.. This is because current implementations of code analysis tools like Revise and JET parse top-level files using their own |
|
Thanks @aviatesk! It's good that it might help solve those kind of issues as well reducing my feeling of "ick something is badly factored" 😅 This API does provide for tooling to be able to run top level evaluation with modified |
Julia's incrementally evaluated top level semantics make it rather tricky to design a lowering interface for top level and module level expressions. Currently these expressions are effectively *interpreted* by eval rather than ever being processed by lowering. However, I'd like a cleaner separation between "low level evaluation" and lowering, such that Core can contain only the low level eval "driver function". I'd like to propose the split as follows: * "Low level" evaluation is about executing a sequence of thunks represented as `CodeInfo` and creating modules for those to be executed inside. * Lowering is about expression processing. In principle, the runtime's view of `eval()` shouldn't know about `Expr` or `SyntaxTree` (or whatever AST we use) - that should be left to the compiler frontend. A useful way to think about the duties of the frontend is to consider the question "What if we wanted to host another language on top of the Julia runtime?". If we can eventually achieve that without ever generating Julia `Expr` then we will have succeeded in separating the frontend. To implement all this I've recast lowering as an incremental iterative API in this change. Thus it's the job of `eval()` to simply evaluate thunks and create new modules as driven by lowering. (Perhaps we'd move this definition of `eval()` over to the Julia runtime before 1.13.) The iteration API is currently oddly bespoke and arguably somewhat non-Julian for two reasons: * Lowering knows when new modules are required, and may request them with `:begin_module`. However `eval()` generates those modules so they need to be passed back into lowering. So we can't just use `Base.iterate()`. (Put a different way, we have a situation which is suited to coroutines but we don't want to use full Julia `Task`s for this.) * We might want to implement this `eval()` in Julia's C runtime code or early in bootstrap. Hence using SimpleVector and Symbol as the return values of `lower_step()` We might consider changing at least the second of these choices, depending on how we end up integrating this into Base.
c42f
force-pushed
the
caf/toplevel-interpret-modules
branch
from
e55da7f to
2b487e1
Compare
|
I figured out a way to make this work on 1.12 so I've updated the PR to include that implementation of I'd note that the iteration API is currently oddly bespoke and arguably somewhat non-Julian (weirdly low level) for two reasons:
We might consider changing the second of these choices, depending on how we end up integrating this into Base. For example, removing the use of |
|
I think I'll merge this for now so I can continue with another PR for the frontend API. I'm fairly happy with the outline but some design details could certainly be revisited as noted above. |
Just as a heads up, the Compiler has grown a That uses a custom workloop embedded in the AbstractInterpreter, so probably not directly applicably to JuliaLowering (and high-complexity anyway), but perhaps worth keeping an eye on to generalize one day. |
Here I've generalized and refactored the incremental lowering code from PR #84, splitting out the interface from the implementation and factoring the parts which should go into `Core` into a `_Core` module. `_Core.eval()` is implemented in terms of a new `_Core.simple_eval()` which can evaluate only `CodeInfo` and the new types `TopLevelCodeIterator`, `BeginModule` and `EndModule`. Evaluation of `TopLevelCodeIterator` is implemented in terms of the incremental lowering interface which must be provided by a compiler frontend. A compiler frontend (subtype of `CompilerFrontend`) must come with implementations of the functions `lower_init` and `parseall` and `lower_step` must be being defined on the return type of `lower_init`. Having a type for the frontend solves the issue of how to tie parsing and lowering together without needing to convert to `Expr`. `parseall(frontend, code, ...)` returns a syntax tree of the preferred expression type for the frontend which can be fed into `lower_init`. This function is just a placeholder - it needs to be generalized to allow `Meta.parse()`, `Meta.parseatom()` and `Meta.parseall()` to be implemented in terms of it. Parser and lowering diagnostics are handled by throwing an exception from the frontend but we may want a fuller diagnostics API instead.
Oooh, interesting! I think the ideal abstraction I'd want in this situation is stackless coroutines. But just doing that by hand with a couple of functions and an iterator type is good enough. |
| if k == K"toplevel" | ||
| push!(iter.todo, (ex, false, 1)) | ||
| return lower_step(iter) | ||
| elseif k == K"module" |
There was a problem hiding this comment.
Need to check for (doc str (module ...))
In general I think the fact that we require module to be a direct child of toplevel (not the usual definition of "at top level") is considered a bug, although not one JuliaLowering needs to fix.
| ex = expand_forms_1(iter.ctx, ex) | ||
| k = kind(ex) | ||
| end | ||
| if k == K"toplevel" |
There was a problem hiding this comment.
toplevel can appear at arbitrary depth in an otherwise-lowerable tree. Is handling surface-level toplevel here worth it when we handle it elsewhere in the general case?
| elseif k == K"module" | ||
| name = ex[1] | ||
| if kind(name) != K"Identifier" | ||
| throw(LoweringError(name, "Expected module name")) |
There was a problem hiding this comment.
Need to check for escaped module name (probably in macro expansion)
…el-interpret-modules Incremental lowering API
4 participants
Julia's incrementally evaluated top level semantics make it rather
tricky to design a lowering interface for top level and module level
expressions. Currently these expressions are effectively interpreted
by eval rather than ever being processed by lowering.
However, I'd like a cleaner separation between "low level evaluation"
and lowering, such that Core can contain only the low level eval "driver
function". I'd like to propose the split as follows:
represented as
CodeInfo, and potentially about creating modules forthose to be executed inside.
In principle, the runtime's view of
eval()shouldn't know aboutExpror
SyntaxTree(or whatever AST we use) - that should be left to thecompiler frontend. A useful way to think about the duties of the
frontend is to consider the question "What if we wanted to host another
language on top of the Julia runtime?". If we can eventually achieve
that without ever generating Julia
Exprthen we will have succeeded inseparating the frontend.
To implement all this I've recast lowering as an incremental iterative
API in this change. Thus it's the job of
eval()to simply evaluatethunks and create new modules as driven by lowering. (Perhaps we'd move
this definition of
eval()over to the Julia runtime before 1.13.)Depends on JuliaLang/julia#59604