Skip to content

Motivation

Jump to bottom
Paul M. Rodriguez edited this page May 29, 2019 · 1 revision

Overlord was not written in a vacuum – I wrote it to solve practical problems. They are far from the only uses for Overlord, but understanding what made Overlord necessary may help you see opportunities to use it.

The context: I wrote Overlord while working on TBRSS, a feed reader. TBRSS was deployed as a single executable, a dumped Clozure Common Lisp image. The Lisp image was monitored, and restarted as necessary, by systemd. For the site, nginx was used as a reverse proxy.

  1. TBRSS had a very comprehensive, very slow test suite. Using Overlord I was able to test only the code that had changed (see Proctor).

  2. The Lisp image had to contain a lot of baked-in data. E.g. block lists for hosts, or HTTPS Everywhere rules. This data had to be baked into the image so restarts would be effectively instantaneous; reloading and parsing data at run-time, possibly under heavy load, was unacceptable. Overlord would fetch, load, and parse the required data sources as part of building the image.

  3. For the site, I needed to maximize caching. In the ideal case pages that were completely static could be pre-rendered and dumped to disk already compressed to be served by nginx without Lisp being involved. Pages that were almost static could be similarly pre-rendered, with the remainder of the page (like login status) filled in by client-side JavaScript. Pages with a limited number of states could have each state fully pre-rendered in memory and served from Lisp as a pre-compressed blob. Some pages, mostly dynamic, still had sections that could be pre-rendered. All this would have been unmanageable by hand.

  4. I wanted to be able to write Lisp files in arbitrary syntaxes. If there were existing tooling for a language – editor modes, linters and checkers, optimizers, etc. – then I wanted to be able to use that tooling, while still compiling to Lisp. For example, I needed to work directly in SQL so I could take advantage of new Postgres features – real SQL, not a DSL, but with the queries being compiled into Lisp. Templates wouldn’t do; I needed the files to be real source files, pre-compiled and tracked as dependencies. This became cl-yesql. (Note that support for languages has been moved out of Overlord into a separate project, Vernacular.)

Clone this wiki locally