Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Don't hold walproposer WAL in memory #141

Merged
merged 4 commits into from
Apr 4, 2022
Merged

Don't hold walproposer WAL in memory #141

merged 4 commits into from
Apr 4, 2022

Conversation

petuhovskiy
Copy link
Member

No description provided.

@petuhovskiy petuhovskiy requested a review from arssher March 4, 2022 23:37
@petuhovskiy petuhovskiy force-pushed the walproposer-wal-on-disk branch from 78e0afb to 50c7fe6 Compare March 22, 2022 14:33
@petuhovskiy petuhovskiy mentioned this pull request Mar 24, 2022
Merged
@petuhovskiy
Copy link
Member Author

This PR is generally ready for review. CI tests are passing, all except test_pageserver_catchup_while_compute_down timeouted in debug, investigating that.

@petuhovskiy
Copy link
Member Author

Looked at the test and the logs, failure is probably not related to this PR. I see that test_pageserver_catchup_while_compute_down test is running queries on compute when pageserver is down, seems that sometimes it's not possible and test hangs.

@arssher arssher mentioned this pull request Mar 28, 2022
Merged
arssher added a commit to neondatabase/neon that referenced this pull request Mar 28, 2022
@arssher
Make shared_buffers large in test_pageserver_catchup.
8bdea32 
We intentionally write while pageserver is down, so we shouldn't query it.

Noticed by @petuhovskiy at
neondatabase/postgres#141 (comment)
arssher added a commit to neondatabase/neon that referenced this pull request Mar 28, 2022
@arssher
Make shared_buffers large in test_pageserver_catchup.
75002ad 
We intentionally write while pageserver is down, so we shouldn't query it.

Noticed by @petuhovskiy at
neondatabase/postgres#141 (comment)
@petuhovskiy petuhovskiy force-pushed the walproposer-wal-on-disk branch from c1a1d25 to 6c7e336 Compare March 29, 2022 07:02
arssher
arssher approved these changes Apr 1, 2022
View reviewed changes
Copy link

@arssher arssher left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. So this commit removes in memory queue, puts during recovery WAL on disk in its usual place and runs separate XLogReader for each safekeeper. This prevents OOM when one safekeeper deeply lags behind. Let's put that into commit message.

To preserve sending of empty AppendRequests to deliver commit_lsn and other metadata to safekeepers now SendAppendRequests always sends one message even if there is no WAL data.

I added minor comments to the branch, consider merging them.

BTW I did a quick performance comparison during normal work and it is strange. With patch
test_walproposer_pgbench.tps_pgbench: 12,308.0612
without
test_walproposer_pgbench.tps_pgbench: 5,390.6744
not sure why yet

@petuhovskiy
Copy link
Member Author

Performance change for a local single safekeeper setup is quite strange, but I expect some difference in a cloud setup. I'll probably try to run my cloud tests after everything is merged to main.

@petuhovskiy petuhovskiy merged commit 8481459 into main Apr 4, 2022
@petuhovskiy petuhovskiy deleted the walproposer-wal-on-disk branch April 4, 2022 12:52
@petuhovskiy
Copy link
Member Author

Here are results for running cloud benchmark after this patch. It's ~20k TPS, seems similar to the previous results from here (~20.5k TPS).

MMeent pushed a commit that referenced this pull request Jul 7, 2022
@petuhovskiy @MMeent
Don't hold walproposer WAL in memory (#141)
7f2e701 
WAL is no longer in memory to prevent OOM in the compute. Removed in-memory queue because it's not needed anymore. When streaming, WAL is now read directly from disk. Every safekeeper has a separate XLogReader. walproposer will now read as much WAL as it can for a single AppendRequest message, it can help with recovering lagging safekeepers. Because Recovery needs to save WAL for streaming, now walproposer can write WAL to disk and `--sync-safekeepers` mode will create pg_wal directory if needed. Replication slot `restart_lsn` is now synced with `truncate_lsn` to prevent truncation of disk WAL until needed.
MMeent pushed a commit that referenced this pull request Aug 18, 2022
@petuhovskiy @MMeent
Don't hold walproposer WAL in memory (#141)
db5ea19 
WAL is no longer in memory to prevent OOM in the compute. Removed in-memory queue because it's not needed anymore. When streaming, WAL is now read directly from disk. Every safekeeper has a separate XLogReader. walproposer will now read as much WAL as it can for a single AppendRequest message, it can help with recovering lagging safekeepers. Because Recovery needs to save WAL for streaming, now walproposer can write WAL to disk and `--sync-safekeepers` mode will create pg_wal directory if needed. Replication slot `restart_lsn` is now synced with `truncate_lsn` to prevent truncation of disk WAL until needed.
lubennikovaav pushed a commit that referenced this pull request Nov 21, 2022
@petuhovskiy @lubennikovaav
Don't hold walproposer WAL in memory (#141)
f1834ee 
WAL is no longer in memory to prevent OOM in the compute. Removed in-memory queue because it's not needed anymore. When streaming, WAL is now read directly from disk. Every safekeeper has a separate XLogReader. walproposer will now read as much WAL as it can for a single AppendRequest message, it can help with recovering lagging safekeepers. Because Recovery needs to save WAL for streaming, now walproposer can write WAL to disk and `--sync-safekeepers` mode will create pg_wal directory if needed. Replication slot `restart_lsn` is now synced with `truncate_lsn` to prevent truncation of disk WAL until needed.
MMeent pushed a commit that referenced this pull request Feb 10, 2023
@petuhovskiy @MMeent
Don't hold walproposer WAL in memory (#141)
5061ad9 
WAL is no longer in memory to prevent OOM in the compute. Removed in-memory queue because it's not needed anymore. When streaming, WAL is now read directly from disk. Every safekeeper has a separate XLogReader. walproposer will now read as much WAL as it can for a single AppendRequest message, it can help with recovering lagging safekeepers. Because Recovery needs to save WAL for streaming, now walproposer can write WAL to disk and `--sync-safekeepers` mode will create pg_wal directory if needed. Replication slot `restart_lsn` is now synced with `truncate_lsn` to prevent truncation of disk WAL until needed.
MMeent pushed a commit that referenced this pull request Feb 10, 2023
@petuhovskiy @MMeent
Don't hold walproposer WAL in memory (#141)
91712dd 
WAL is no longer in memory to prevent OOM in the compute. Removed in-memory queue because it's not needed anymore. When streaming, WAL is now read directly from disk. Every safekeeper has a separate XLogReader. walproposer will now read as much WAL as it can for a single AppendRequest message, it can help with recovering lagging safekeepers. Because Recovery needs to save WAL for streaming, now walproposer can write WAL to disk and `--sync-safekeepers` mode will create pg_wal directory if needed. Replication slot `restart_lsn` is now synced with `truncate_lsn` to prevent truncation of disk WAL until needed.
MMeent pushed a commit that referenced this pull request May 11, 2023
@petuhovskiy @MMeent
Don't hold walproposer WAL in memory (#141)
c699884 
WAL is no longer in memory to prevent OOM in the compute. Removed in-memory queue because it's not needed anymore. When streaming, WAL is now read directly from disk. Every safekeeper has a separate XLogReader. walproposer will now read as much WAL as it can for a single AppendRequest message, it can help with recovering lagging safekeepers. Because Recovery needs to save WAL for streaming, now walproposer can write WAL to disk and `--sync-safekeepers` mode will create pg_wal directory if needed. Replication slot `restart_lsn` is now synced with `truncate_lsn` to prevent truncation of disk WAL until needed.
tristan957 pushed a commit that referenced this pull request Aug 10, 2023
@petuhovskiy @tristan957
Don't hold walproposer WAL in memory (#141)
573dfe5 
WAL is no longer in memory to prevent OOM in the compute. Removed in-memory queue because it's not needed anymore. When streaming, WAL is now read directly from disk. Every safekeeper has a separate XLogReader. walproposer will now read as much WAL as it can for a single AppendRequest message, it can help with recovering lagging safekeepers. Because Recovery needs to save WAL for streaming, now walproposer can write WAL to disk and `--sync-safekeepers` mode will create pg_wal directory if needed. Replication slot `restart_lsn` is now synced with `truncate_lsn` to prevent truncation of disk WAL until needed.
tristan957 pushed a commit that referenced this pull request Nov 8, 2023
@petuhovskiy @tristan957
Don't hold walproposer WAL in memory (#141)
63d87dc 
WAL is no longer in memory to prevent OOM in the compute. Removed in-memory queue because it's not needed anymore. When streaming, WAL is now read directly from disk. Every safekeeper has a separate XLogReader. walproposer will now read as much WAL as it can for a single AppendRequest message, it can help with recovering lagging safekeepers. Because Recovery needs to save WAL for streaming, now walproposer can write WAL to disk and `--sync-safekeepers` mode will create pg_wal directory if needed. Replication slot `restart_lsn` is now synced with `truncate_lsn` to prevent truncation of disk WAL until needed.
tristan957 pushed a commit that referenced this pull request Nov 8, 2023
@petuhovskiy @tristan957
Don't hold walproposer WAL in memory (#141)
d7ac667 
WAL is no longer in memory to prevent OOM in the compute. Removed in-memory queue because it's not needed anymore. When streaming, WAL is now read directly from disk. Every safekeeper has a separate XLogReader. walproposer will now read as much WAL as it can for a single AppendRequest message, it can help with recovering lagging safekeepers. Because Recovery needs to save WAL for streaming, now walproposer can write WAL to disk and `--sync-safekeepers` mode will create pg_wal directory if needed. Replication slot `restart_lsn` is now synced with `truncate_lsn` to prevent truncation of disk WAL until needed.
tristan957 pushed a commit that referenced this pull request Feb 5, 2024
@petuhovskiy @tristan957
Don't hold walproposer WAL in memory (#141)
1aacb3f 
WAL is no longer in memory to prevent OOM in the compute. Removed in-memory queue because it's not needed anymore. When streaming, WAL is now read directly from disk. Every safekeeper has a separate XLogReader. walproposer will now read as much WAL as it can for a single AppendRequest message, it can help with recovering lagging safekeepers. Because Recovery needs to save WAL for streaming, now walproposer can write WAL to disk and `--sync-safekeepers` mode will create pg_wal directory if needed. Replication slot `restart_lsn` is now synced with `truncate_lsn` to prevent truncation of disk WAL until needed.
tristan957 pushed a commit that referenced this pull request Feb 5, 2024
@petuhovskiy @tristan957
Don't hold walproposer WAL in memory (#141)
4618b4d 
WAL is no longer in memory to prevent OOM in the compute. Removed in-memory queue because it's not needed anymore. When streaming, WAL is now read directly from disk. Every safekeeper has a separate XLogReader. walproposer will now read as much WAL as it can for a single AppendRequest message, it can help with recovering lagging safekeepers. Because Recovery needs to save WAL for streaming, now walproposer can write WAL to disk and `--sync-safekeepers` mode will create pg_wal directory if needed. Replication slot `restart_lsn` is now synced with `truncate_lsn` to prevent truncation of disk WAL until needed.
tristan957 pushed a commit that referenced this pull request Feb 6, 2024
@petuhovskiy @tristan957
Don't hold walproposer WAL in memory (#141)
87dae2a 
WAL is no longer in memory to prevent OOM in the compute. Removed in-memory queue because it's not needed anymore. When streaming, WAL is now read directly from disk. Every safekeeper has a separate XLogReader. walproposer will now read as much WAL as it can for a single AppendRequest message, it can help with recovering lagging safekeepers. Because Recovery needs to save WAL for streaming, now walproposer can write WAL to disk and `--sync-safekeepers` mode will create pg_wal directory if needed. Replication slot `restart_lsn` is now synced with `truncate_lsn` to prevent truncation of disk WAL until needed.
tristan957 pushed a commit that referenced this pull request May 10, 2024
@petuhovskiy @tristan957
Don't hold walproposer WAL in memory (#141)
0c4a55e 
WAL is no longer in memory to prevent OOM in the compute. Removed in-memory queue because it's not needed anymore. When streaming, WAL is now read directly from disk. Every safekeeper has a separate XLogReader. walproposer will now read as much WAL as it can for a single AppendRequest message, it can help with recovering lagging safekeepers. Because Recovery needs to save WAL for streaming, now walproposer can write WAL to disk and `--sync-safekeepers` mode will create pg_wal directory if needed. Replication slot `restart_lsn` is now synced with `truncate_lsn` to prevent truncation of disk WAL until needed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@arssher arssher arssher approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@petuhovskiy @arssher