libpqwp_async_read should PQgetCopyData before PQconsumeInput #6055
Labels
c/storage/safekeeper
Component: storage: safekeeper
Comments
arssher
added a commit
that referenced
this issue
Dec 9, 2023
It is similar to XLogReader, but when either requested segment is missing locally or requested LSN is before basebackup_lsn NeonWALReader asynchronously fetches WAL from one of safekeepers. Patch includes walproposer switch to NeonWALReader, splitting wouldn't make much sense as it is hard to test otherwise. This finally removes risk of pg_wal explosion (as well as slow start time) when one safekeeper is lagging, at the same time allowing to recover it. In the future reader should also be used by logical walsender for similar reasons (currently we download the tail on compute start synchronously). The main test is test_lagging_sk. However, I also run it manually a lot varying MAX_SEND_SIZE on both sides (on safekeeper and on walproposer), testing various fragmentations (one side having small buffer, another, both), which brought up #6055 closes #1012
arssher
added a commit
that referenced
this issue
Dec 14, 2023
To avoid a lot of redundant memmoves and bloated input buffer. fixes #6055
arssher
added a commit
that referenced
this issue
Dec 14, 2023
To avoid a lot of redundant memmoves and bloated input buffer. fixes #6055
arssher
added a commit
that referenced
this issue
Dec 15, 2023
It is similar to XLogReader, but when either requested segment is missing locally or requested LSN is before basebackup_lsn NeonWALReader asynchronously fetches WAL from one of safekeepers. Patch includes walproposer switch to NeonWALReader, splitting wouldn't make much sense as it is hard to test otherwise. This finally removes risk of pg_wal explosion (as well as slow start time) when one safekeeper is lagging, at the same time allowing to recover it. In the future reader should also be used by logical walsender for similar reasons (currently we download the tail on compute start synchronously). The main test is test_lagging_sk. However, I also run it manually a lot varying MAX_SEND_SIZE on both sides (on safekeeper and on walproposer), testing various fragmentations (one side having small buffer, another, both), which brought up #6055 closes #1012
arssher
added a commit
that referenced
this issue
Dec 15, 2023
To avoid a lot of redundant memmoves and bloated input buffer. fixes #6055
arssher
added a commit
that referenced
this issue
Dec 26, 2023
It is similar to XLogReader, but when either requested segment is missing locally or requested LSN is before basebackup_lsn NeonWALReader asynchronously fetches WAL from one of safekeepers. Patch includes walproposer switch to NeonWALReader, splitting wouldn't make much sense as it is hard to test otherwise. This finally removes risk of pg_wal explosion (as well as slow start time) when one safekeeper is lagging, at the same time allowing to recover it. In the future reader should also be used by logical walsender for similar reasons (currently we download the tail on compute start synchronously). The main test is test_lagging_sk. However, I also run it manually a lot varying MAX_SEND_SIZE on both sides (on safekeeper and on walproposer), testing various fragmentations (one side having small buffer, another, both), which brought up #6055 closes #1012
arssher
added a commit
that referenced
this issue
Dec 26, 2023
To avoid a lot of redundant memmoves and bloated input buffer. fixes #6055
arssher
added a commit
that referenced
this issue
Dec 26, 2023
It is similar to XLogReader, but when either requested segment is missing locally or requested LSN is before basebackup_lsn NeonWALReader asynchronously fetches WAL from one of safekeepers. Patch includes walproposer switch to NeonWALReader, splitting wouldn't make much sense as it is hard to test otherwise. This finally removes risk of pg_wal explosion (as well as slow start time) when one safekeeper is lagging, at the same time allowing to recover it. In the future reader should also be used by logical walsender for similar reasons (currently we download the tail on compute start synchronously). The main test is test_lagging_sk. However, I also run it manually a lot varying MAX_SEND_SIZE on both sides (on safekeeper and on walproposer), testing various fragmentations (one side having small buffer, another, both), which brought up #6055 closes #1012
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Like libpqrcv_receive. Otherwise we bloat input buffer and make it slow by constant memmoves if walproposer is slow to consume WAL stream; can be tested by setting low MAX_SEND_SIZE buf on rust size and running recovery. Found while testing
#5948
e.g.
time pytest --capture=no -k "test_late_init[debug-pg16"with insert size increases in this test.
The text was updated successfully, but these errors were encountered: