automatic layer eviction

[problame]

automatic layer eviction

This patch adds a per-timeline periodic task that executes an eviction
policy. The eviction policy is configurable per tenant.

Two policies exist:
- NoEviction (the default one)
- LayerAccessThreshold

The LayerAccessThreshold policy examines the last access timestamp per
layer in the layer map and evicts the layer if that last access is
further in the past than a configurable threshold value.
This policy kind is evaluated periodically at a configurable period.
It logs a summary statistic at `info!()` or `warn!()` level, depending
on whether any evictions failed.

This feature has no explicit killswitch since it's off by default.

---

Try out with neon_local by putting the following into pageserver.toml

[tenant_config]
eviction_policy = { kind = "LayerAccessThreshold" , period = "2s", threshold = "10s" }

[problame]

NB: the stats reset feature conflicts with this one: if we continuously reset the stats, then the eviction loop has no last access date to base the decision on. Need to figure out what to do here, reset is quite useful in #3540

[koivunej]

I think this is looking good! So the minimum value for unused is the periodic e2e check interval, and ... That's it? What's missing for this to be put very aggressively into staging?

Well, could move all of the eviction stuff into tenant/timeline/eviction.rs for example.

[problame]

The last pushes should address all review comments.

I'll rebase this PR on top of #3557 once that's merged.

[problame]

NB: the stats reset feature conflicts with this one: if we continuously reset the stats, then the eviction loop has no last access date to base the decision on. Need to figure out what to do here, reset is quite useful in #3540

Addressed this by duplicating the LayerAccessStatsInner.
See commit
maintain stats twice, once for api, once for eviction policy

Well, could move all of the eviction stuff into tenant/timeline/eviction.rs for example.

Good idea, did that.

[problame]

Need to figure out how to implement tenant config update via API before we can merge this.

[koivunej]

First round of questions, minor suggestions.

[problame]

For some reason GitHub doesn't show a comment box for some of your comments, so, I'll reply here:

So this is json within json, why? Aah, unsure if I've ever used this neon_local functionlity. Does seem weird but what are the alternatives?

Well, I don't want to spend time on implementing a CLI-friendly representation of the eviction policy variants here. IMO there should be a serde_cli crate that standardizes such stuff.

Any objections with going forward with JSON?

[problame]

Force-push: squash & proper commit message (updated top-level PR description as well)