soroban-rpc: Review and improve memory consumption

[2opremio]

Soroban-rpc uses some in-memory data structures for recent data (LedgerBucketWindow Go datastructure).

It would be good to know how much memory soroban-rpc needs when running in pubnet (average and peak).

While working on #553 I noticed that, when bootstrapping the in-memory datastructures, we read all the txmeta from the database at once:

https://github.com/stellar/soroban-tools/blob/a210572b64a91b862ec3190bba627ea0503282c2/cmd/soroban-rpc/internal/daemon/daemon.go#L130-L143

This worries me, since, when running on pubnet, the DB contains ~20GB worth of txmeta. Probably resulting in a memory consumption north of 20GB.

Note that this will add on top of the captive-core memory consumption.

We should consider

• Streaming the txmeta in batches when bootstrapping.
• Reducing the time window we use for the in-memory data structures.
• Last resort: move some of the in-memory datastructures to the on-disk DB.

In the context of this issue, we should also re-evaluate the amount of memory effectively used by the in-memory transaction store. Currently, the default is set to 2 hours ( 1440 ledgers ). We might want to revise the default based on this analysis.

[stellarsaur]

#904 Addresses the main memory consumption issue - now we stream ledgers instead of load them all at once from the database.

The other main areas of memory usage are:

• Captive-core
• LedgerBucketWindow

As long as we have a bounded retention window for transactions and events, memory usage shouldn't be an issue. Note that we retain 2 hours of ledgers in the transaction store, and 24 hours of ledgers in the events store. If anything, we could consider reducing the retention window for the events store to something like 12 hours of ledgers, but it probably won't make a substantial difference. Also, we could consider reducing the retention window for the transactions store to something like 1 hour of ledgers, but once again, it probably won't make a substantial difference unless the ledgers were huge.

Finally, preliminary load tests outlined here didn't reveal any issues with memory usage running against testnet and futurenet.

All in all, I would consider this issue solved. We can always reopen if memory usage becomes problematic in the future.

[stellarsaur]

Based on an internal discussion, rather than decreasing the transaction or event ledger windows, we actually may want to increase the transaction ledger window to 24 hours, since it's what users expect.

Closing this but we can reopen it in the future if memory usage becomes an issue.