track per-vat XS heap (snapshot) sizes in our mainnet logging dashboard

[warner]

What is the Problem Being Solved?

We have two mainnet vats which are currently very large, as measured by the size of their xsnap worker heaps. v7-board is about 582MB, and v43-walletFactory is about 625MB. And both are growing.

Large xsnap heaps take a long time to perform garbage collection on, because the entire object graph must be painted by the "mark" half of the mark-and-sweep process. This causes the bringOutYourDead deliveries to take longer (because they force a GC pass), and causes random other deliveries to take longer (when they happen to trigger "organic" GC). They also take a long time to serialize into a "heap snapshot" (the save-snapshot action we do every 200 deliveries), and then to load back in again afterwards.

I discovered this problem (and the growth rate) by manually inspecting the logs and the DB snapshots I make each time I reboot my personal follower node. We need this kind of growth rate to be more visible to more people.

So I'd like to see the XS heap/snapshot size of all vats included in our mainnet-status dashboards.

Description of the Design

There are two ways to measure this size. The first is to ask the linux kernel about the VmSize of all the xsnap-worker processes. This tracks the actual linux process' memory space size. I think we might already be tracking this (it's a pretty common SRE-type thing to track). On the plus side, it is updated constantly (no lag). On the minus side, it is noisy (this number includes any memory-mapped files, and memory that is allocated by the process but not yet used by the JS engine). Sometimes we track VmRSS, which excludes memory that the process hasn't touched recently, but that's even noisier, because memory pressure from unrelated processes will evict pages and make VmRSS look artificially low.

The other is to query the SwingStore DB for the uncompressedSize of the most recent heap snapshot. On the plus side, this is directly tied to one of our cost metrics (the time it takes to serialize/write/compress/reload the heap snapshot), and is a pretty good proxy for the other cost metric (GC time, which is linear in the number of Slot entries plus the size of the Chunk entries that provide backing-store for Arrays and other reference-bearing slabs). On the minus side, it lags a bit (the DB only has the size of the latest heap snapshot, and we only create one every 200 deliveries).

The following DB query will extract the latest heap-snapshot sizes:

% sqlite3 -readonly -box swingstore.sqlite 'SELECT vatID,uncompressedSize FROM snapshots WHERE inUse=1'

The -readonly prevents this sqlite3 command from obtaining even a read-lock on the DB, to avoid interfering with a running swingset kernel. I've accidentally crashed my follower by doing a lengthy DB query against a live swingstore: the kernel will retry for a few hundred milliseconds when it tries to obtain the write-lock, but if it can't get one in time, it will crash.

This doesn't need to be sampled any faster than once per minute. I'm most interested in growth over the timespan of hours or days.

It would be extra cool to somehow annotate the graph with vat upgrade events.

Security Considerations

None at this level, any follower can get this information, and the XS heap size should be same for all validators (since the heap snapshot is in-consensus).

Scaling Considerations

The query should be pretty fast: there are only a few hundred vats, and we have an index on (vatID,inUse). So as long as we're using -readonly I'm not worried about its impact on operations.

CC @toliaqat

[warner]

It would be great to also include alerts when the growth rate exceeds some threshold. Once we fix v43-walletFactory and v7-board, we should look to see what other growth remains, and either fix it or use it as a warning threshold. There are legitimate reasons why some small growth might continue to occur (e.g. if we have more outstanding contract sessions at any given time because of increased chain activity), but in general any collection or object which is expected to have high cardinality should be stored in virtual/durable objects or collections, and not consuming heap space. So most growth indicates something surprising at best, and something that threatens chain stability at worst.