Add new sync method to support #617 using txid poller to fix #656 #811
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a sync mechanism specific for authoritative deployments for providing standby and query-only secondaries. Particularly when mixing the two.
With #617, there is a lot of potential data in PostgreSQL that will not be included in NRTM. NRTM was always a poor solution for standby instances, with
nrtm_access_list_unfiltered
as a hack on top, and potential loss of journal data and any suppressed objects. PostgreSQL replication fixes a lot of these issues, including even retaining serials. However, preloaded data will go out of date on standby instances. Redis replication is not a full fix, as some of the preloaded data is in memory in worker processes.PostgreSQL does support triggers on a replica as noted in #656, but not when using WAL streaming, which is the only reasonable option here, as logical streaming breaks sequences and seems to have issues with upserts. The idea of monitoring for records where
updated
is higher than the last check is insufficient, because it does not catch row deletions or certain suppression state changes.Best option:
select timestamp from pg_last_committed_xact()
. This does not allow us to filter for object types. However, considering route(6) and soon as-set are preloaded, many transactions will require pre-load store updates. There will therefore be additional overhead, but at the benefit of always being sure preloaded data gets updated. Caveat: the timestamp is not database-specific, so does not work well when running multiple databases. But these seems like an acceptable cost for people who need to run hot standby servers and is a fairly solid fix for the long standing difficulties in these setups.The suppression status will be replicated as well, i.e. the
!f
queries will also work and suppressed objects are not lost during a switchover. However, the local config state shown in!J
may not be consistent. This needs to be reflected in the docs. Also, during switchover, which requires a restart anyways, PGP keys have to be reimported into the local keychain. Hard and not worthwhile to do while running as standby.