Fix [MQB]: mqbc::StorageMgr: Transition to available only when all primary active #416
Conversation
kaikulimu
force-pushed
the
fsm-test--available-only-active-primary
branch
3 times, most recently
from
80f71b6
to
e5d59b2
Compare
|
|
||
| pinfo.setPrimaryStatus(value); | ||
| if (bmqp_ctrlmsg::PrimaryStatus::E_ACTIVE == value) { | ||
| d_fileStores[partitionId]->setPrimary(pinfo.primary(), |
There was a problem hiding this comment.
why is this in setPrimaryStatusForPartition and not in setPrimaryForPartition?
There was a problem hiding this comment.
FileStore should only set primary after the primary becomes active. The FileStore is unable to work with a passive primary. I will rename the function as FileStore::setActivePrimary to make the point clear.
| << " primary, this advisory could " | ||
| << "be from the true one. Will" | ||
| << " buffer the advisory for now."; | ||
| d_storageManager_p->bufferPrimaryStatusAdvisory(primaryAdv, |
There was a problem hiding this comment.
Question, does pinfo.primaryNode() get assigned upon PrimaryStatusAdvisory or in the FSM flow there is another trigger? We receive PrimaryStatusAdvisory and we do not have partition primary, why not assign the primary then?
There was a problem hiding this comment.
In FSM mode, the source of truth for partition assignments are in the cluster state snapshot of the CSL file. As part of healing, a new leader assigns partitions and then applies the assignments in its first CSL advisory. Primary status adviosries can be stale; that's why we have a lot of checks in this function in the first place. My original idea was to simply ignore all primary status advisories and purely rely upon FSM for partition assignments. However, FSM can heal a replica but neglect to set a primary as active. Thus, I came up with the idea of buffering primary status advisories. If an advisory is not stale (i.e. matching primary node and leaseId), then we trust the availability advisory.
![bmq-oss-ci[bot]](https://avatars.githubusercontent.com/u/1416818?s=60&v=4){kind=small}
kaikulimu
force-pushed
the
fsm-test--available-only-active-primary
branch
from
835ba43
to
0116bdf
Compare
|
@dorjesinpo Back to you |
kaikulimu
force-pushed
the
fsm-test--available-only-active-primary
branch
2 times, most recently
from
06c92b2
to
337db61
Compare
Signed-off-by: Yuan Jing Vincent Yan <yyan82@bloomberg.net>
Signed-off-by: Yuan Jing Vincent Yan <yyan82@bloomberg.net>
Signed-off-by: Yuan Jing Vincent Yan <yyan82@bloomberg.net>
Signed-off-by: Yuan Jing Vincent Yan <yyan82@bloomberg.net>
Signed-off-by: Yuan Jing Vincent Yan <yyan82@bloomberg.net>
kaikulimu
force-pushed
the
fsm-test--available-only-active-primary
branch
from
337db61
to
8059ba6
Compare
Fixes two flaky integration tests in FSM mode:
In
test_basicoftest_restart.py, there was an issue where a replica could advertise availability before all primaries are active; then, a proxy could repoen queue and post message with no avail. The fix is to transition to available only when all primaries are active.In
test_kill_post_startoftest_strong_consistency.py, there was an issue where replicas are not issuing receipts to the primary after restart. This was because healing replicas in FSM mode were not buffering primary status advisories to process later, and thus not setting the correct primary in the FileStore. After I added the buffering logic, the tests pass.