swingstore mode to move/archive historical transcript spans to separate files, outside the DB

[warner]

What is the Problem Being Solved?

As discussed here, I kinda want a swing-store mode like { keepTranscripts: false, archiveTranscriptsInto: DIRNAME }, which would instruct the transcriptStore to write a compressed span into a new file in DIRNAME, just before deleting the items of that span.

My goal is to enable an external script (some combination of mv and scp or s3cmd) to move the completed files out of the way, upload or transfer them to some cheaper location, and then delete the originals. I want to save space on my follower's SSD, but I still want to have those old transcripts around for analysis purposes.

In addition to the new swing-store option and implementation, it would also need some plumbing through the app.toml config file (similar to what #10032 is doing).

Description of the Design

Security Considerations

None, this is a write-only option for swing-store, to a directory specified by the host application (which also specifies the directory for the SQLite file).

Scaling Considerations

This might improve performance of the SQLite transcriptItems table, in configurations where historical spans are not being retained, by drastically (90%) reducing the number of rows, and correspondingly reducing the size of the index. However I don't have any reason to believe this is currently consuming a significant amount of time in our system.

The stored files would be compressed, which would save about 90% of the disk space, compared to uncompressed items remaining in SQLite. However that's not a fair comparison, because either you're retaining historical transcripts (keepTranscripts: true) in which case they stay in SQLite and you're consuming a whole lot of disk, or you're allowing them to be pruned from the DB (keepTranscripts: false) in which case you're saving all of the space. This third option that I'm asking for will consume more overall space than plain keepTranscripts: false, but at least it will only be consuming about 10% of the disk space that the uncompressed transcriptItems items would have needed. Also note that #8318 would achieve the all-but-10% space savings (by virtue of compression) while still retaining all the historical spans.

Test Plan

unit tests in packages/swing-store, plus something (not sure what) for the app.toml wiring

Upgrade Considerations

None, when this feature is first enabled on a node, subsequent span rollovers will start populating the given directory. This is a local, not-in-consensus property, like maxVatsOnine or the option to retain historical spans at all.

To avoid losing data, a node which wishes to use this mode should change from { keepTranscripts: true } to { keepTranscripts: false, archiveTranscriptsInto: DIRNAME } in a single step, and never run the kernel with merely { keepTranscripts: false }.

cc @gibson042 @mhofman

[mhofman]

just before deleting the items of that span

IMO this option should unconditionally save them there, whether they would be deleted or not by keepTranscripts.

If not too much work, please also consider adding an equivalent archiveHeapSnapshotsInto, writing at the same time as inserted in the DB (aka just a tee). That would likely balloon the disk usage, but would be helpful for some debugging.