storage: investigate larger default range sizes

[jeffrey-xiao]

#16954 was a major blocker in being able to increase the default range size from 64 MiB (another potential problem is #22348). Since #16954 has been closed, we can experiment with larger default range sizes.

I ran tpcc on a 3 node cluster with 16 vCPUs, 2500 warehouses, and an average range size of 500 MiB and a max range size of 1 GiB and nothing seemed to blow up.

There were a couple of latency spikes during the ramp-up period of tpcc, but I think they were caused by the destruction of old replicas when the initial splits were merged from setting up the workload (not too sure about this).

[petermattis]

Not a blocker, but I recall there is a timeout for snapshot send operations. If we increase the range size significantly, we may need to increase that timeout. For future proofing, it may be a good idea to adjust the timeout based on the size of the snapshot. Or since we send the snapshot in chunks, put the timeout on the sending of each chunk instead of the operation as a whole.

[ajwerner]

Not a blocker, but I recall there is a timeout for snapshot send operations.

This timeout isn't directly on the snapshot itself but rather is due to the timeout on queue processing in the storage package. I'm going to type a patch to allow queues to independently control that timeout. For the raft snapshot queue my sense is maybe it'd be best to set the queue process timeout quite high and then do what you suggest with a timeout per chunk or per snapshot.

[ajwerner]

Err we have the ability to set the timeout per queue but it was set to the default 1m.

[ajwerner]

Concerns:

• Snapshot timeouts
  • fixed by storage: make queue timeouts dynamic and add cluster setting for snapshots #42686
• Backup memory usage
  • Backups generate SSTs which are range sized. These SSTs are built in C++ and handed back to go. A 512MB slice is a big slice in memory constrained environments.
• CDC catch up scans
  • CDC uses export requests to pull large amounts of data to perform catch up scans. We need a mechanism to limit the size which is retrieved.
• RESTORE
  • there might also be a mem blowup on the RESTORE side since all the SSTs that cover a given span being restored need to be downloaded and opened as iterators that can be wrapped up in one merging-iterator from which to produce the RESTORE’d span SST. SST index is at the end, so we have to download the whole thing before we can create an iterator over it. Right now that means all he SSTs that contain keys for a given span being imported are held in ram while we construct, in ram, the resulting SST that covers that range. We could spill them to disk but encryption at rest makes that somewhat more involved.

[dt]

Another reason RESTORE might prefer not to spill to disk is that disk bandwidth is often a hot commodity during a RESTORE or bulk-ingestion -- using any of it to write something other than the final ingestion SSTs data could potentially make RESTORE slower.

[ajwerner]

@dt correct me if I'm wrong but from the discussion on slack it seems like we can mitigate all of the above concerns if we creates SSTs during backup with some target size rather than as the entire range size.

SSTs created prior to increasing the range size will target 64MB. If we continued to target 64MB exported SSTs for BACKUP then that would continue to be the size used for RESTORE. The potential downside of this approach is that RESTORE will create smaller ranges. We're not too concerned about this downside in the long run because the ranges will get merged together. The problem with waiting for the merge queue to merge the ranges together is that it will need to move data around to collocate the ranges for merging which could lead to a bunch of extra data movement following the RESTORE. @dt does limiting the size of the SSTs created during backup seem challenging?