Horizon Lite: Automate ledger cache management for the web service.

[sreuland]

A crucial part of getting Horizon Lite to an acceptable level of performance is caching ledgers locally to avoid latency spikes from downloading ledgers.

There's a toolkit to "pre-load" the cache in addition to the existing behavior of caching ledgers on use, e.g.:

exp/lighthorizon> ./lighthorizon cache build --start <ledger>

This same process should be done in parallel on startup of the web service if a parameter is provided, and it should subsequently be integrated into the k8s deployment.

Acceptance Criteria

• An operator can easily control how much index data is pre-cached locally based on number of ledgers as a web server startup parameter. Enable parallel cache warm up, web server should be responsive as early as possible, using the cache immediately, of which it's state can be changing as it's warmed up from async population process which does not interfere with concurrent web read access attempts to the index.

Sub tasks

• enable index to support concurrent read/write operations safely, @Shaptic - support/storage: Make the on-disk cache thread-safe. #4575
• add --preload-cache=<count> flag support to web server startup params, k8s manifest, @sreuland

[sreuland]

@Shaptic , wanted to confirm on scope, this would be one new startup parameter to the horizon light web server, like --preload-cache=true|false and the behavior is that web server will launch an async parallel routine that executes the ./tools cache build --start <ledger> , and can it just pass 0 for --start to default to earliest ledger in the source index?

[Shaptic]

That's a great question 🤔 I don't think starting at the earliest available ledger is a good idea. Based on our research, recent ledgers make much more sense to cache up-front. Perhaps we can do something like --preload-cache=<count> which does a cache build --start $(latest - count) --count <count> (where latest is a pseudo-placeholder for the latest ledger in the index [or meta? this is hard to determine from the index] store).

As for async/parallel, my perspective was more that it happens before the web server launches, i.e. if you want to deploy the webserver, it first loads X ledgers into the cache, then starts the webserver. So it's almost part of the provisioning/installation/setup step of deployment. I'm concerned about synchronization and race conditions issues if it's done in parallel to serving requests.

[paulbellamy]

I think (if possible) it would be better for it to happen in the background after the webserver is launched. That way you can launch quickly, and still have a hot cache. Though that might be more complicated, as you would be serving requests so you'd want those cache fills to take precendent so they don't get evicted. If the operator is concerned about cache latency they could use a ramp-up or delay config option in their load-balancer (I forget what HAProxy calls this). But IMO it is better to be serving requests slowly than refusing until the cache is populated (which may or may not even help, depending on the queries).

As for what to populate initially, for the default we use 90 days of ledgers. so maybe the latest 90 days? or maybe even just the latest 30 days, then start serving requests? Depending on cache size it could be cache_max / 3 of the latest ledgers?

[2opremio]

If you want to do it at the start, k8s supports init containers https://kubernetes.io/docs/concepts/workloads/pods/init-containers/

This means that as long as the tool is containerized, you don’t need do any any further scripting or embed the command in horizon light itself

[sreuland]

@Shaptic , any thoughts on lightweight/fast approaches for getting the 'latest' indexed ledger from the index store. I see the explorer tool index stats file:///tmp/indices could get the latest checkpoint ledger from index store, but it iterates every account first, so that seems expensive for this purpose. My first thought was to expand the scope here to change index processor to emit a 'latest' file at root directory of index store, it just contains one line with greatest sequence number in the index store, then retrieval is O(1), as single file read op.

[Shaptic]

You're right, @sreuland, it's pretty much impossible without changes to the index builder (hence the necessity for the tool). I think the initial design meant that the ledger source and the index source were "in sync," but otherwise it's not feasible without additional metadata.