compiler: unify the backend processing

[zerbina]

Summary

Right now, each code generator / backend works differently. Both cgen and jsgen use a recursive approach, where the code for a procedure is generated when it's first used. While cgen emits the generated code into multiple C files (one corresponding to each NimSkull module), jsgen emits all of it into a single file, and also requires special handling for inner procedures (lambda lifting is disabled for the JS backend).

For the VM, code generation works significantly different: the code generator (vmgen) is, for the most part, only responsible for generating code. Invoking the code generator to generate the bytecode for all alive procedures is left to the callsite. During compile-time execution, this is the responsibility of the JIT logic (vmjit), and for the VM backend it's that of vmbackend. The latter uses, supported by vmgen, an iterative approach for discovering all alive procedures and passing them to the code generator.

The problem with the recursive approach is that it's inflexible: how discovery of alive procedures happens is an intrinsic property of it and can't be easily changed. In addition, transforming a procedure's code and applying the MIR passes to it has to happen from inside the code generators, meaning that they have to carry the necessary state around, further complicating the whole implementation.

This PR implements a facility for collecting the AST of the whole program and applying the pre-processing (e.g. transf, applying eligible MIR passes) to all code passed to it, making the resulting procedures accessible as a stream.

The approach is an evolution of how the processing is implemented in #424 (which itself evolved from vmbackend), with the difference that it's more general and that the procedures are, except for inner closure procedures, only processed one-by-one instead of all at once. This change makes the layer more flexible, allowing it to be used with both interleaved and non-interleaved compilation (interleaved here meaning: alternating between semantic analysis and code generation).

The C, JS, and VM code generators are now no longer responsible for discovering dependencies, and, in the case of cgen/jsgen, processing them. Both things are now implemented by a dedicated orchestrator (which used the aforementioned processing facilities) for each backend, similar to vmbackend.

For the first revision, the orchestrators are only concerned with procedures, but will eventually also manage code generation for constants and globals.

---

Notes for reviewers

• the PR is an early proof-of-concept. The focus so far was only to make it work
• once finalized, this PR will only include the new processing facilities -- changing each code generator to make use of them will all be separate PRs
• removing the IC backend would make the cgen transition easier

[zerbina]

There were a few issues with the previous handling of procedure-level globals (in this PR):

1. entities (currently only routines) used by their initialization logic were queued from inside routine processing, foregoing the callsite
2. MIR passes were not applied to their initialization logic (a pre-existing issue)
3. processing a single procedure has to mutate/track state unrelated to procedures (e.g. a module's pre-initialization code fragment)

The problem with 1. is that the callsite might want to know about dependencies (the C backend does) -- queuing procedures from inside a call to next meant that it couldn't, breaking inline handling for the C backend, for example.

Number 3 is a severe layering and single-responsibility violation that required next to mutate module-related state, and ProcedureIter to know about destructors for globals.

The solution

The fix is to use a much simpler and more local approach: store the extracted globals and the MIR code fragment containing their initialization logic in the Procedure object returned from next. This way, the callsite is responsible for generating the destruction logic, building up a module's pre-initialization code, and queuing dependencies.

In the future, it might make sense to perform the lifting of procedure-level globals into a special section of a routine during semantic analysis.

Clean-up for globals

Another issue was with how globals requiring destruction (via a destructor call) were/are handled (both in this PR and previously). For top-level globals, the injectdestructors pass generates a statement for calling the respective destructor (if one is available) and appends it to the globalDestructors statement list stored as part of ModuleGraph. cgen and jsgen then query said statement list when closing the main module, generate code for it, and append the result to the end of the main module's init procedure.

There are multiple problems with this approach:

1. the injectdestructors pass has side-effects on pass-external state (i.e. it modifies the ModuleGraph)
2. order of destruction for globals is tied to the order in which the code their defined in is transformed
3. given that a significant number of routines in the compiler have access to the ModuleGraph instance, globalDestructors is essentially a global

The first one would become a practical problem once the injectdestructors pass is also run for code meant for compile-time execution, as destructor calls for globals only existing in a compile-time context are then added to the same statement list used for normal globals.

To get rid of globalDestructors, module structs are introduced. A module struct encapsulates all data owned by a module. Currently, these are globals (both top-level and procedure-level ones) and .threadvars. All entities that make up a module's struct are collected, and, once all alive code is processed, the resulting information used to generate the clean-up logic.

Right now, the structs only exist at a logical level, but it might make sense to also emit them as real types in the generated code in the future.

[zerbina]

The PR is starting to become too large, both in terms of code changes and impact. I'm going to split the changes here into multiple smaller, separate PRs.

[zerbina]

Superseded by #712, #714, and, most importantly, #777. Splitting this PR up turned out to be the right choice; the implementation I have now ended up with is much cleaner and simpler.