Perform inmem_smgr cleaup after processing each record

[knizhnik]

No description provided.

[hlinnaka]

So, I realized that this isn't quite right either. Imagine that num_temp_buffers == 4, and you have a WAL replay function that does this:

target_buf = XLogReadBufferForRedo(<Target page>);
<apply changes on the target page>
MarkBufferDirty(target_buf);
UnlockReleaseBuffer(target_buf);
other_buf1 = XLogReadBufferForRedo(<Target page>);
other_buf2 = XLogReadBufferForRedo(<Target page>);
other_buf3 = XLogReadBufferForRedo(<Target page>);
other_buf4 = XLogReadBufferForRedo(<Target page>);

This will cause the target page to be evicted from the buffer cache, and stored in the "inmem_smgr". After replaying the record, we will call smgrinit(), which will throw away the target page and the changes that were applied to it.

So we need to prevent the target page from being thrown away. I see two options:

1. hack the page eviction code in localbuf.c to never evict the redo target buffer, or
2. hack inmem_smgr.c to not throw away the target page

[hlinnaka]

I tried to change zenith_wal_redo.c so that it keeps the target page pinned, from BeginRedoForBlock() until the GetPage() call. That would give a tiny speedup too, by avoiding the hash lookup in GetPage() and PushPage(). But that didn't work, because some WAL redo functions take the "cleanup lock" on the page, which requires that there are no other pins on the page.

[knizhnik]

This will cause the target page to be evicted from the buffer cache, a
I do not think that it is possible.
Please notice that XLogReadBufferForRedo ignores all records except target records and only for it returns BLK_NEEDS_REDO. Actually it can not get content of this page: smgr is transient and only one page image is received from page server. So redo can only construct new pages. But it is only done when XLogReadBufferForRedo returns BLK_NEEDS_REDO. So I do not think that target page can be somehow evicted from buffers and pinning is actually neded.
But we can add some asserts here...

[hlinnaka]

This will cause the target page to be evicted from the buffer cache, a

I do not think that it is possible.
Please notice that XLogReadBufferForRedo ignores all records except target records and only for it returns BLK_NEEDS_REDO. Actually it can not get content of this page: smgr is transient and only one page image is received from page server. So redo can only construct new pages. But it is only done when XLogReadBufferForRedo returns BLK_NEEDS_REDO. So I do not think that target page can be somehow evicted from buffers and pinning is actually neded.
But we can add some asserts here...

Well, in RBM_ZERO_AND_LOCK mode it is possible, at least.

[knizhnik]

Well, in RBM_ZERO_AND_LOCK mode it is possible, at least.

I see.
Unfortunately I do not know better solution than just patch bufmgr to prohibit eviction of such page.
See my proposed implementation. Do not like it but...

[hlinnaka]

We use the local buffer manager in the WAL redo process, so this needs to be in localbuf.c

[knizhnik]

Shame on me:(
Yet another attempt.

[hlinnaka]

Feel free to take over neondatabase/neon#1573 to test this, btw.

[knizhnik]

So the tests are passed. What's then?

[hlinnaka]

Let's squash and merge this