Compaction causes full reimaging

[bojanserafimov]

Steps to reproduce

I haven't locally reproduced it, but I saw it happen in prod 9 times on the same project, odd-brook-545806. I used the timeline visualizer to spot the problem, highlighted in red

Expected result

Compaction makes L1 layers, which trigger image creation only where needed.

Actual result

Three little L1 layers cause the entire database to get covered in new image layers. As a result, we have 9 full unnecessary copies of the database.

Also this bug complicates perf testing, since reused projects will behave differently than fresh projects.

Environment

prod

Why is this happening?

We take an image when there are at least 3 L1 layers over the latest image. But every compaction creates 2 very long L1 layers that span all the way to the left and right of the keyspace, because there are some nonrel keys in those parts of the keyspace that get updated often.

Proposal for solution

Instead of creating images, include these long and sparse delta layers in the next compaction round. Basically treat them as L0 layers. Redefine the definition of L1 to be "a sufficiently dense layer, regardless of how many times it's been compacted".

[knizhnik]

And what do you think about my "partial image layers" approach (#2563) ?
It also doesn't suffer from this problem and doesn't require compaction (reshuffling) at all.

[bojanserafimov]

And what do you think about my "partial image layers" approach

Do you have a text explanation that goes with the presentation? I get the concept of "delta with page images", but I'm missing a lot of specifics on changes to heuristics. Do we still take regular images after x deltas, per partition?

[knizhnik]

And what do you think about my "partial image layers" approach

Do you have a text explanation that goes with the presentation? I get the concept of "delta with page images", but I'm missing a lot of specifics on changes to heuristics. Do we still take regular images after x deltas, per partition?

It seems to me that I have answered this question somewhere (may be in Slack?).
With partial image layers there is no need to create full image layers frequently, because there are two main reasons for generation of full image layers:

1. Restrict number of applied deltas by walredo.
2. Make it possible to garbage collect old layers.
   With partial images walredo is not needed and GC can be performed relatively rare taken in account current PiTR settings (several days).