Support Baseline Resync

[yuwmao]

For Nuraft baseline resync, we separate the process into two layers: HomeStore layer and Application layer.
We use the first bit of the obj_id to indicate the message type: 0 is for HS, 1 is for Application.

In the HomeStore layer, leader needs to transmit the DSN to the follower, this is intended to handle the following case:

• Leader sends snapshot at LSN T1 to follower F1.
• F1 fully receives the snapshot and now at T1.
• Leader yield its leadership, F1 elected as leader.
  In this sequence the incremental resync will not kicked in to update the m_next_dsn, and as result, duplication may occur.

[JacksonYao287]

if I understand correctly , this pr aims to update the m_next_dsn in the snapshot receiver side, right?

for the receiver , after baseline resync is completed, incremental resync will start, where new log entries will be recevied and committed. according to the code here

HomeStore/src/lib/replication/repl_dev/raft_repl_dev.cpp

Lines 933 to 934 in 6a2dfd8

	m_next_dsn.compare_exchange_strong(cur_dsn, rreq->dsn() + 1);
	}

m_next_dsn will be updated by handle_commit (committing the received log entry), and thus m_next_dsn will eventually be up-to-date as leader. so , I am not sure is it necessary to add a separate step to sync m_next_dsn.

pls correct me if I am wrong or miss something

[xiaoxichen]

We discussed this case in the review,

The case is,

• Leader sends snapshot at LSN T1 to follower F1.
• F1 fully receives the snapshot and now at T1.
• Leader yield its leadership, F1 elected as leader.

In this sequence the incremental resync will not kicked in to update the m_next_dsn.

Yes now it only update the m_next_dsn but more generically , it is a channel for baseline resync at HS level.

[JacksonYao287]

Leader yield its leadership, F1 elected as leader.

yes , theoretically , this will happen. moreover , we need to trigger a cp_flush after baseline resync is done to guarantee m_next_dsn is persisted to metaservice to prevent this from happening even if F1 restarts. cc @yuwmao

[yuwmao]

Leader yield its leadership, F1 elected as leader.

yes , theoretically , this will happen. moreover , we need to trigger a cp_flush after baseline resync is done to guarantee m_next_dsn is persisted to metaservice to prevent this from happening even if F1 restarts. cc @yuwmao

Ok, in our design, the last obj is an empty message, let's call cp_flush in the last message.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 34.88372% with 28 lines in your changes missing coverage. Please review.

Project coverage is 66.54%. Comparing base (1a0cef8) to head (0cb90b8).
Report is 97 commits behind head on master.

Files with missing lines	Patch %	Lines
src/lib/replication/repl_dev/raft_repl_dev.cpp	30.76%	18 Missing ⚠️
...rc/lib/replication/repl_dev/raft_state_machine.cpp	37.50%	10 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@             Coverage Diff             @@
##           master     #596       +/-   ##
===========================================
+ Coverage   56.51%   66.54%   +10.03%     
===========================================
  Files         108      109        +1     
  Lines       10300    10810      +510     
  Branches     1402     1476       +74     
===========================================
+ Hits         5821     7194     +1373     
+ Misses       3894     2906      -988     
- Partials      585      710      +125     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[xiaoxichen]

Leader yield its leadership, F1 elected as leader.

yes , theoretically , this will happen. moreover , we need to trigger a cp_flush after baseline resync is done to guarantee m_next_dsn is persisted to metaservice to prevent this from happening even if F1 restarts. cc @yuwmao

Ok, in our design, the last obj is an empty message, let's call cp_flush in the last message.

I would suggest do that in apply_snapshot(), we also update lsn there.

[yuwmao]

Leader yield its leadership, F1 elected as leader.

yes , theoretically , this will happen. moreover , we need to trigger a cp_flush after baseline resync is done to guarantee m_next_dsn is persisted to metaservice to prevent this from happening even if F1 restarts. cc @yuwmao

Ok, in our design, the last obj is an empty message, let's call cp_flush in the last message.

I would suggest do that in apply_snapshot(), we also update lsn there.

Ok, make sense.

[yamingk]

A general comment: is there a plan to add unit test case (with flip) for the error path, such as read_snapshot_obj, apply_snapshot_obj, save_snapshot_obj, etc to verify the retry logic will be happening in the baseline resync flow?
It is okay to do it with another PR.