Significantly improve performance, reliability and memory usage

[arohner]

Significantly improve compile time, CPU usage and memory usage. Significant rework of internals.

The previously observed memory "leak" was not a leak. To avoid race conditions and to get parallelism, we were creating a classloader per thread and compiling in every thread. This wasn't leaking memory, just using more memory than a dev laptop has available. Wall clock time was also poor because reloading the same classes once per core (8-10x) is inefficient.

We instead use a single classloader shared by all threads. We avoid the previous need to discard a classloader (causing more wasted work!) by instead reloading downstream dependencies, the same way a human would at a REPL. Parallelism is handled via agents. We create an agent for each namespace in the project, and send the commands to it. Before compiling a namespace, we send require to all of its dependents. If the namespace contains deftype, definterface or defprotocol, we send reload to all downstream dependencies, to avoid AOT hell.

[arohner]

Oh, for posterity: this has known issues with potemkin, manifold and byte-streams. Those libraries use the funky definterface+ et. al, which contain

(defmacro definterface+ [name & body]
  (when-not (resolve name)
    `(definterface ~name ~@body)))

If the namespace is already present on the classpath, calling compile on this code will not output .class files to be placed in the jar. The problem is exacerbated by bazel sometimes compiling jars twice, once for compile-time deps, and once for runtime. The two solutions are: use the griffinbank/* forks of these libraries, or in the future, use bazel's path-mapping feature to avoid the double compile.

[miridius]

first pass complete, 4 files left to review, will finish them later today

[miridius]

Is there a risk that this will cause some things to successfully compile even though they depend on something no longer in the classpath?

[miridius]

first pass complete, 4 files left to review, will finish them later today

oh yeah I finished this yesterday but didn't have any more comments to add :)

[miridius]

looking much cleaner now! very nice

[miridius]

👍

[miridius]

since we're never using the agent state, we're using it to do to 3 things:

1. run thunks one at a time per ns
2. run thunks in different nses in parallel
3. wait for those to be done

Could we:

1. create a lock per ns
2. require and compile would be (future (locking ns-lock (do-stuff)))
3. deref the returned futures

[miridius]

^ lock + future works if we don't care about ordering of the submitted requests, which maybe we don't (because of sha?). But if we do, I think a better idea (also mentioned in Slack) is to create a (Executors/newSingleThreadExecutor) per ns and then (.submit ns-exec ^Runnable (bound-fn [] ...)). that will return a future that you can deref to await

[arohner]

This moved from banksy's deps.edn because any require on the source tree can trigger it. The proper way to fix is to move it into a bazel toolchain, but that's a lot more work