Load DSL compilers and extensions ahead of time in add-on mode

[vinistock]

Motivation

While profiling DSL generation in the add-on, I noticed we were spending a lot of time running Dir.glob. The code path that lead to invoking glob multiple times was loading DSL compilers and extensions, which invokes both glob directly and Gem.find_files (which invokes glob under the hood).

There's no point in reloading DSL compilers and extensions every single time. We can load it once and then the forked process will have all of the expected classes loaded already.

Implementation

We need to use the workspace path declared by the editor to load all of this (which may not match Dir.pwd), so I made a new notification in the server add-on. It gets invoked once after boot.

The other side of the change is to stop loading the compilers and extensions in the regular DSL path, which how we can save some time.

[vinistock]

• Load DSL compilers and extensions ahead of time in add-on mode #2152 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[paracycle]

Do we need this extra complexity? Can we not make the loader be idempotent and refuse to do any loading if called a second time? That wouldn't require any knowledge about LSP mode or anything.

[vinistock]

Do we need this extra complexity? Can we not make the loader be idempotent and refuse to do any loading if called a second time? That wouldn't require any knowledge about LSP mode or anything.

I indeed would prefer that approach, but there are several layers of why that's not possible with the current approach. Instead of decoupling the CLI from the core of Tapioca, so that we could have two frontends (CLI + add-on), we are invoking the CLI by reloading it.

That approach was causing severe memory bloat because loading the CLI multiple times is not in any way idempotent. To move forward with the prototype, we started running DSL in a fork to avoid polluting the memory of the main process - which fixes the bloat, but also means that we lose all of the state between runs.

A simple thing like this wouldn't work because the separate runs won't share state.

def load
  return if @loaded

  # ...

  @loaded = true
end