Add lazy_lsru_download flag to compute spec and automatically make decision about lazy SLRU download

[knizhnik]

Problem

Lazy SLRU download was implemented and merged long ago.
But it relies on lazy_slru_download flag in tenant config, disabled by default.
It is assumed that somebody will monitor Neon metrics, detect clients with slow startup time and if it is caused by huge
size of SLRUs (most likely CLOG or multixasts), then set this flag for this tenant.

But it actually never happen.

The idea of this PR is automatically detect when lazy download of SLRUs may be needed (when they are large).
We get size of SLRUs we first need to extract them. But extraction from KV storage itself takes most of the time of node startup, not the time needed to send basebackup through network. So it has not sense to check SLRUs size during each basebackup. Instead of it we should do it once (if lazy SLRU download was not already switched on for this tenant) and stored in timeline. So if SLRUs are really large, then we will not extract them and include in basebackup next time.

Summary of changes

1. Add ComputeFeature::LazySlruDownload feature flag to ComputeSpec allowing to able or disable this fauture for the particular region.
2. Add --lazy-slru-download options basebackup command
3. Add lazy_slru_download_threshold parameter to page server config, allowing it to automatically make decision whether to include or not LSRUs in basebackup.

So the logic now is the following:
If lazy SLRU download logic is enabled by control plane (which includes correspond option in base backup),
then page server will look if lazy_slru_download flag is set for the particular timeline or tenant.
If it is not set, then it includes SLRU in backup.
But we also check if size of some of any SLRUs exceeds lazy_slru_download_threshold and update
lazy_slru_download flag for this timeline.

Please notice that lazy_slru_download flag is stored only in in-memory timeline metadata, so after PS restart it is lost and PS has to recheck size of SLRUs once again.

[github-actions]

7744 tests run: 7366 passed, 0 failed, 378 skipped (full report)

---

Flaky tests (5)

Postgres 17

• test_deletion_queue_recovery[validate-keep]: release-x86-64-with-lfc
• test_migration_to_cold_secondary: release-x86-64-with-lfc, release-arm64-with-lfc
• test_timeline_copy[500000]: release-x86-64-with-lfc

Postgres 15

• test_sharding_compaction[4-32768-True]: release-arm64-with-lfc

Code coverage* (full report)

• functions: 32.8% (8645 of 26370 functions)
• lines: 48.6% (73262 of 150643 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
7c5ae84 at 2025-02-27T11:09:47.612Z :recycle:}

[hlinnaka]

blocks contains the blocks of just the current partition, part. That doesn't seem right.

[knizhnik]

Thank you
I had wrong assumption that partition=SLRU kind.
Fixed

[hlinnaka]

In the long term, do you think we should switch completely to lazy SLRU download, and never include them in the basebackup? If so, I suggest we simplify this. Let's drop the threshold in the pageserver, and have just the compute feature flag. Then roll it out to everyone.

If we want to still include SLRU segments in the basebackup when they're not too large, then I'd suggest that we do that slightly differently. A threshold makes sense, but I'd suggest that we always include e.g. the latest 1 MB of clog segments or last segment of all SLRUs or something like that. It doesn't have to be all-or-nothing.

[knizhnik]

In the long term, do you think we should switch completely to lazy SLRU download, and never include them in the basebackup? If so, I suggest we simplify this. Let's drop the threshold in the pageserver, and have just the compute feature flag. Then roll it out to everyone.

No, I do no think so.
For most tenants size of SLRU is really small and it is better to include them in basebackup and avoid extra network roundtrips.

[VladLazar]

I like the general approach.

Some thoughts:

• I think we should check slru size before serving first base-backup (see Add lazy_lsru_download flag to compute spec and automatically make decision about lazy SLRU download #11012 (comment))
• We should update the pageserver_basebackup_query_seconds metric with a lazt_slru_download label. Otherwise, it's hard to reason about how fast basebackups actually are.

[VladLazar]

nit: The total size is already tracked in SlruSegmentBuilder. You could make SlruSegmentBuilder::finish return it instead of tracking it here again.

[knizhnik]

Just think more about @hlinnaka second proposal - always include last segment of SLRU in basebackup.
My first though was that it is not so good idea, because in most cases Postgres doesn't need to access CLOG - it relies either on record hints, either on snapshot horizon.

But then I understand that to be able to update SRLU, we in any case need first to read updated segment (in most cases - last segment). So it may be really useful and reduce real startup time.

But then I recognised that there is not efficient way in current KV storage API to locate last segment (@VladLazar please correct me). So to locate last segment we in any case have to iterate through all segments. Which makes this optimization senseless.

My second thought was that there are many other pages which will be requested by backend in any case - some tables from system catalog. But we do not have such optimisation now and access this catalog pages in usual way.

Then I return back to first @hlinnaka proposal: always use on-demand download. If we are doing it for much more pages of system catalog, then doing it for few SLRU segments should not somehow significantly affect startup time.

Also, may be my first thought was also not so stupid: yes we need to download SLRU segment in order to update it. But we need to update SLRU if we perform some updates (write transaction). But most queries to the database are read-only, do not need XID and do not write something to SLRU. So using lazy download in this we can really reduce startup time even for small SLRUs (not sure how noticeable it will be).

So @hlinnaka , @VladLazar what do you think about removing all this logic with lazy SLRU download threshold and use it always (when it is enabled by control plane? Also may be remove it from tenant config?

[VladLazar]

Caching this after the first basebackup leads to strange load pattern on the pageserver.

I think it would be better to check the SLRU size before serving the first basebackup.
You can do that by getting the size of Timeline::get_slru_keyspace. It reads the slru segment directory for each slru kind and then one read per segment to get the size, so should be fairly cheap. Might even be cheap enough that we can do it every time without caching.