soroban-rpc: increase transaction retention window to 24 hours

[tsachiherman]

What?

Requested by product - increase transaction retention window to 24 hours. This would reduce confusion about retention window.

Before increasing the retention window, we should evaluate what the expected memory utilization is going to be and re-align hardware specifications as needed.

Why?

24 hour retention: should be a buffer that fills up uniformly in terms of parity for events + ledger data (not 24 hours for somethings and 2 hours for others). So if we can confirm through testing this is supportable, we should do it (with some margin of safety, eg, enough memory/disk for 24hours of data + some safety margin).

We are testing this empirically, and that will inform the work here. It seems likely to be fine, given that current testnet instances are using < 1GB RAM. However, it's important to test this vs pubnet, which has a different load profile than testnet.

Work

1. Confirm the memory usage of 24 hours of transaction retention on pubnet.
2. In discussion with product, decide if that memory usage has acceptable hardware requirements for users.
3. If acceptable, implement this as a single window, so that there will be a single value going forward.
4. Update documentation to indicate the change from 2 hour window to 24 hour window.
5. Operator guidance: Provide a default config of 24 hours with guidance in our Runbook showing how to adjust this window if necessary.

[tsachiherman]

@mollykarcher I was asked to add this to the next release; would that be possible for P0 ?

[tomerweller]

Just to clarify: this is the default value, operators can still configure a different value, right?

Regardless, I suggest having a single configuration value for retention window for both transactions and events

[mollykarcher]

Just to clarify: this is the default value, operators can still configure a different value, right?

Correct, this is just the default value.

Regardless, I suggest having a single configuration value for retention window for both transactions and events

So this sounds a little different than the title/description here, but I'd agree that this is what would actually reduce the user confusion with it. We want to align the retention windows, not simply increase the default transaction retention window. So I think that would mean we'd remove both transaction-retention-window and event-retention-window configs, and replace those with a single retention-window that controls the retention for both.

@mollykarcher I was asked to add this to the next release; would that be possible for P0 ?

I'm in favor. If we go with what I've described above, we'd be removing configuration options which is technically a breaking change, so I'd prefer to get it out before the stable release. It also sounds like minimal work, assuming the memory footprint doesn't take a bit hit. Either way, I think getting it into the stable release is definitely doable.

[ire-and-curses]

and re-align hardware specifications as needed.

To be clear: we need to understand the expected memory utilisation and then decide if that is acceptable for hardware requirements.

I've updated the ticket body to cover this, and to describe the product rationale, because in the future we might want to remember why we did this.

[tsachiherman]

I'm going to add another task here : update the documentation to reflect this change.

[tsachiherman]

With stellar/stellar-cli#1063 merged in, I think it would be safe to have a 16GB machine running a pubnet node with a 24 hour transaction history.

[tsachiherman]

I have not worked on the in-memory events store, but expect that a similar gains could be duplicated there.

[mollykarcher]

@stellarsaur can you run the same profiling you did for RPC with the updated transaction window (to 24 hours), so we can get a sense of how this will affect the memory consumption?

[stellarsaur]

I ran Soroban RPC for 24 hours on a dev box with the updated transaction window to 24 hours on both Futurenet and Testnet. In both cases, I noted the base memory consumption using htop to be about 450Mb after starting Soroban RPC and catching up ingestion. After 24 hours, the memory consumption was about 550Mb and there was no increase in CPU usage. Therefore, we would be using an extra ~100Mb of memory on Futurenet and Testnet.

Caveat here is that there will be more transactions in Pubnet, so memory consumption is most likely to increase more based on that increased volume. However, this is a change that is pretty easy to roll back in case there is an issue (immediate fix for users could be to change the command line flag for the retention window to be smaller while we revert). We could load test by expanding this PR to send actual transactions close to what we're currently receiving, but not entirely sure it's necessary for this.

Example for Futurenet:
sudo docker run -p 8001:8001 -p 8000:8000 -v /home/tylersanderson/config:/config stellar/soroban-rpc:20.0.0-rc3-39 --captive-core-config-path="/config/captive-core-futurenet.cfg" --captive-core-storage-path="/var/lib/stellar/captive-core" --captive-core-use-db=true --stellar-core-binary-path="/usr/bin/stellar-core" --db-path="/var/lib/stellar/soroban-rpc-db.sqlite" --stellar-captive-core-http-port=11626 --friendbot-url="https://friendbot-futurenet.stellar.org/" --network-passphrase="Test SDF Future Network ; October 2022" --history-archive-urls="http://history-futurenet.stellar.org" --admin-endpoint="0.0.0.0:8001" --endpoint="0.0.0.0:8000" --event-retention-window=17280 --transaction-retention-window=17280