Allow nextgenrepl to real-time replicate reaps #6
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is to address the issue of reaping across sync'd clusters. Without this feature it is necessary to disable full-sync whilst independently replicating on each cluster. Now if reaping via riak_kv_reaper the reap will be replicated assuming the `riak_kv.repl_reap` flag has been enabled. At the receiving cluster the reap will not be replicated any further. There are some API changes to support this. The `find_tombs` aae_fold will now return Keys/Clocks and not Keys/DeleteHash. The ReapReference for riak_kv_repaer will now expect a clock (version vector) not a DeleteHash, and will also now expect an additional boolean to indicate if this repl is a replication candidate (it will be false for all pushed reaps). The object encoding for nextgenrepl now has a flag to indicate a reap, with a special encoding for reap references.
ThomasArts
reviewed
Sep 12, 2023
Clarify specs
ThomasArts
reviewed
Sep 13, 2023
ThomasArts
approved these changes
Sep 13, 2023
stephenwood24-nhs
approved these changes
Sep 14, 2023
martinsumner
added a commit
that referenced
this pull request
Nov 14, 2023
* Allow nextgenrepl to real-time replicate reaps This is to address the issue of reaping across sync'd clusters. Without this feature it is necessary to disable full-sync whilst independently replicating on each cluster. Now if reaping via riak_kv_reaper the reap will be replicated assuming the `riak_kv.repl_reap` flag has been enabled. At the receiving cluster the reap will not be replicated any further. There are some API changes to support this. The `find_tombs` aae_fold will now return Keys/Clocks and not Keys/DeleteHash. The ReapReference for riak_kv_repaer will now expect a clock (version vector) not a DeleteHash, and will also now expect an additional boolean to indicate if this repl is a replication candidate (it will be false for all pushed reaps). The object encoding for nextgenrepl now has a flag to indicate a reap, with a special encoding for reap references. * Update riak_object.erl Clarify specs * Take timestamp at correct point (after push) * Updates following review * Update rebar.config
martinsumner
added a commit
that referenced
this pull request
Nov 14, 2023
* Allow nextgenrepl to real-time replicate reaps This is to address the issue of reaping across sync'd clusters. Without this feature it is necessary to disable full-sync whilst independently replicating on each cluster. Now if reaping via riak_kv_reaper the reap will be replicated assuming the `riak_kv.repl_reap` flag has been enabled. At the receiving cluster the reap will not be replicated any further. There are some API changes to support this. The `find_tombs` aae_fold will now return Keys/Clocks and not Keys/DeleteHash. The ReapReference for riak_kv_repaer will now expect a clock (version vector) not a DeleteHash, and will also now expect an additional boolean to indicate if this repl is a replication candidate (it will be false for all pushed reaps). The object encoding for nextgenrepl now has a flag to indicate a reap, with a special encoding for reap references. * Update riak_object.erl Clarify specs * Take timestamp at correct point (after push) * Updates following review * Update rebar.config
Merged
martinsumner
added a commit
that referenced
this pull request
Feb 13, 2024
* Merge pull request #1 from nhs-riak/nhse-contrib-kv1871 KV i1871 - Handle timeout on remote connection * Trigger batch correctly at each size (#4) * Force timeout to trigger (#3) Previously, the inactivity timeout on handle_continue could be cancelled by a call to riak_kv_rpelrtq_snk (e.g. from riak_kv_rpelrtq_peer). this might lead to the log_stats loop never being triggered. * Configurable %key query on leveled (#8) Can be configured to ignore tombstone keys by default. * Allow nextgenrepl to real-time replicate reaps (#6) * Allow nextgenrepl to real-time replicate reaps This is to address the issue of reaping across sync'd clusters. Without this feature it is necessary to disable full-sync whilst independently replicating on each cluster. Now if reaping via riak_kv_reaper the reap will be replicated assuming the `riak_kv.repl_reap` flag has been enabled. At the receiving cluster the reap will not be replicated any further. There are some API changes to support this. The `find_tombs` aae_fold will now return Keys/Clocks and not Keys/DeleteHash. The ReapReference for riak_kv_repaer will now expect a clock (version vector) not a DeleteHash, and will also now expect an additional boolean to indicate if this repl is a replication candidate (it will be false for all pushed reaps). The object encoding for nextgenrepl now has a flag to indicate a reap, with a special encoding for reap references. * Update riak_object.erl Clarify specs * Take timestamp at correct point (after push) * Updates following review * Update rebar.config * Make current_peers empty when disabled (#10) * Make current_peers empty when disabled * Peer discovery to recognise suspend and disable of sink * Update src/riak_kv_replrtq_peer.erl Co-authored-by: Thomas Arts <thomas.arts@quviq.com> * Update src/riak_kv_replrtq_peer.erl Co-authored-by: Thomas Arts <thomas.arts@quviq.com> --------- Co-authored-by: Thomas Arts <thomas.arts@quviq.com> * De-lager * Add support for v0 object in parallel-mode AAE (#11) * Add support for v0 object in parallel-mode AAE Cannot assume that v0 objects will not happen - capability negotiation down to v0 on 3.0 Riak during failure scenarios * Update following review As ?MAGIC is distinctive constant, then it should be the one on the pattern match - with everything else assume to be convertible by term_to_binary. * Update src/riak_object.erl Co-authored-by: Thomas Arts <thomas.arts@quviq.com> --------- Co-authored-by: Thomas Arts <thomas.arts@quviq.com> * Update riak_kv_ttaaefs_manager.erl (#13) For bucket-based full-sync `{tree_compare, 0}` is the return on success. * Correct log macro typo --------- Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is to address the issue of reaping across sync'd clusters. Without this feature it is necessary to disable full-sync whilst independently replicating on each cluster.
Now if reaping via riak_kv_reaper the reap will be replicated assuming the
riak_kv.repl_reapflag has been enabled. At the receiving cluster the reap will not be replicated any further.There are some API changes to support this. The
find_tombsaae_fold will now return Keys/Clocks and not Keys/DeleteHash. The ReapReference for riak_kv_repaer will now expect a clock (version vector) not a DeleteHash, and will also now expect an additional boolean to indicate if this repl is a replication candidate (it will be false for all pushed reaps).The object encoding for nextgenrepl now has a flag to indicate a reap, with a special encoding for reap references.