From 920c9738efbe9b1028d6846c79cdcd5a1c668771 Mon Sep 17 00:00:00 2001 From: Evgeniy Osintsev Date: Wed, 20 Apr 2022 16:53:07 +0300 Subject: [PATCH 1/3] Add raft-related constants in tables and initial iproto_raft request description Part of #2814 --- doc/dev_guide/internals/box_protocol.rst | 47 ++++++++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/doc/dev_guide/internals/box_protocol.rst b/doc/dev_guide/internals/box_protocol.rst index 75476b429c..9c44346ac6 100644 --- a/doc/dev_guide/internals/box_protocol.rst +++ b/doc/dev_guide/internals/box_protocol.rst @@ -70,6 +70,9 @@ The IPROTO constants that identify requests that we will mention in this section IPROTO_ROLLBACK=0x10 IPROTO_RAFT_CONFIRM=0x28 IPROTO_RAFT_ROLLBACK=0x29 + IPROTO_RAFT=0x1e + IPROTO_RAFT_PROMOTE=0x1f + IPROTO_RAFT_DEMOTE=0x20 IPROTO_PING=0x40 IPROTO_JOIN=0x41 IPROTO_SUBSCRIBE=0x42 @@ -132,6 +135,10 @@ The IPROTO constants that appear within requests or responses that we will descr IPROTO_FIELD_IS_NULLABLE=0x03 IPROTO_FIELD_IS_AUTOINCREMENT=0x04 IPROTO_FIELD_SPAN=0x05 + IPROTO_RAFT_TERM=0x00 + IPROTO_RAFT_VOTE=0x01 + IPROTO_RAFT_STATE=0x02 + IPROTO_RAFT_VCLOCK=0x03 To denote message descriptions we will say ``msgpack(...)`` and within it we will use modified `YAML `_ so: |br| @@ -1406,6 +1413,46 @@ IPROTO_FLAG_COMMIT (0x01) will be set if this is the last message for a transact IPROTO_FLAG_WAIT_SYNC (0x02) will be set if this is the last message for a transaction which cannot be completed immediately, IPROTO_FLAG_WAIT_ACK (0x04) will be set if this is the last message for a synchronous transaction. +.. _box_protocol-raft: + +IPROTO_RAFT = 0x1e +~~~~~~~~~~~~~~~~~~ + +[TBD] intro + +IPROTO_RAFT рассылается броадкастами всем подключённым к ноде репликам. +Если подразумевается какой-то ответ на запрос (например, когда это запрос другим нодам на голосование), +то нужно, чтобы нода сама была подключена репликой к серверу, от которого ждёт ответ, потому что он придёт по каналу репликации. + +Другими словами, связь между всеми должна быть в обе стороны (фуллмеш) + + +[TBD] update request structure + +.. cssclass:: highlight +.. parsed-literal:: + + # + msgpack(:samp:`{{MP_UINT unsigned integer = size(
) + size()}}`) + #
+ msgpack({ + IPROTO_REQUEST_TYPE: IPROTO_RAFT, + IPROTO_SYNC: :samp:`{{MP_UINT unsigned integer}}`, + IPROTO_INSTANCE_UUID: :samp:`{{uuid}}`, + IPROTO_CLUSTER_UUID: :samp:`{{uuid}}`, + }) + # + msgpack({ + IPROTO_RAFT_TERM=0x00 + IPROTO_RAFT_VOTE=0x01 + IPROTO_RAFT_STATE=0x02 + IPROTO_RAFT_VCLOCK=0x03 + }) + +[TBD] update about response + +Это асинхронный протокол, и response там - другое сообщение IPROTO_RAFT, а бывает вообще без response + .. _box_protocol-illustration: Examples From 3ed729b47c73ee90a7e2e1e6c8eababe17a1f2f2 Mon Sep 17 00:00:00 2001 From: Evgeniy Osintsev Date: Mon, 25 Apr 2022 10:44:17 +0300 Subject: [PATCH 2/3] Update the description of the iproto_raft request Part of #2814 --- doc/dev_guide/internals/box_protocol.rst | 31 ++++++++---------------- 1 file changed, 10 insertions(+), 21 deletions(-) diff --git a/doc/dev_guide/internals/box_protocol.rst b/doc/dev_guide/internals/box_protocol.rst index 9c44346ac6..fc1a2a3e70 100644 --- a/doc/dev_guide/internals/box_protocol.rst +++ b/doc/dev_guide/internals/box_protocol.rst @@ -1418,16 +1418,10 @@ IPROTO_FLAG_WAIT_ACK (0x04) will be set if this is the last message for a synchr IPROTO_RAFT = 0x1e ~~~~~~~~~~~~~~~~~~ -[TBD] intro - -IPROTO_RAFT рассылается броадкастами всем подключённым к ноде репликам. -Если подразумевается какой-то ответ на запрос (например, когда это запрос другим нодам на голосование), -то нужно, чтобы нода сама была подключена репликой к серверу, от которого ждёт ответ, потому что он придёт по каналу репликации. - -Другими словами, связь между всеми должна быть в обе стороны (фуллмеш) - - -[TBD] update request structure +The IPROTO_RAFT request is broadcasted to all the replicas connected to the node. +If there should be a response, for example, in case of a vote request to other nodes, the response will be also an IPROTO_RAFT message. +In this case, the node should be connected as a replica to another node from which the response is expected because the response is sent via the replication channel. +In other words, there should be a full-mesh connection between the nodes. .. cssclass:: highlight .. parsed-literal:: @@ -1437,22 +1431,17 @@ IPROTO_RAFT рассылается броадкастами всем подкл #
msgpack({ IPROTO_REQUEST_TYPE: IPROTO_RAFT, - IPROTO_SYNC: :samp:`{{MP_UINT unsigned integer}}`, - IPROTO_INSTANCE_UUID: :samp:`{{uuid}}`, - IPROTO_CLUSTER_UUID: :samp:`{{uuid}}`, + IPROTO_REPLICA_ID: :samp:`{{MP_INT integer}}`, # ID of the replica which the request came from + }) # msgpack({ - IPROTO_RAFT_TERM=0x00 - IPROTO_RAFT_VOTE=0x01 - IPROTO_RAFT_STATE=0x02 - IPROTO_RAFT_VCLOCK=0x03 + IPROTO_RAFT_TERM: :samp:`{{MP_UINT unsigned integer}}`, # RAFT term of the instance + IPROTO_RAFT_VOTE: :samp:`{{MP_UINT unsigned integer}}`, # Instance vote in the current term (if any). + IPROTO_RAFT_STATE: :samp:`{{MP_UINT unsigned integer}}`, # Instance state; one of the three numbers: 1---follower, 2---candidate, 3---leader. + IPROTO_RAFT_VCLOCK: :samp:`{{MP_ARRAY {{MP_INT SRV_ID, MP_INT SRV_LSN}, {MP_INT SRV_ID, MP_INT SRV_LSN}, ...}}}` # Current vclock of the instance. Presents only on the instances in the "candidate" state (IPROTO_RAFT_STATE == 2). }) -[TBD] update about response - -Это асинхронный протокол, и response там - другое сообщение IPROTO_RAFT, а бывает вообще без response - .. _box_protocol-illustration: Examples From 0ac549277accdf95fd7530da478c77b00e06e992 Mon Sep 17 00:00:00 2001 From: Evgeniy Osintsev Date: Mon, 25 Apr 2022 16:19:08 +0300 Subject: [PATCH 3/3] Add corrections after review Part of #2814 --- doc/dev_guide/internals/box_protocol.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/doc/dev_guide/internals/box_protocol.rst b/doc/dev_guide/internals/box_protocol.rst index fc1a2a3e70..c00dd05c8c 100644 --- a/doc/dev_guide/internals/box_protocol.rst +++ b/doc/dev_guide/internals/box_protocol.rst @@ -1418,8 +1418,10 @@ IPROTO_FLAG_WAIT_ACK (0x04) will be set if this is the last message for a synchr IPROTO_RAFT = 0x1e ~~~~~~~~~~~~~~~~~~ -The IPROTO_RAFT request is broadcasted to all the replicas connected to the node. -If there should be a response, for example, in case of a vote request to other nodes, the response will be also an IPROTO_RAFT message. +A node broadcasts the IPROTO_RAFT request to all the replicas connected to it when the RAFT state of the node changes. +It can be any actions changing the state, like starting a new election, bumping the term, voting for another node, becoming the leader, and so on. + +If there should be a response, for example, in case of a vote request to other nodes, the response will also be an IPROTO_RAFT message. In this case, the node should be connected as a replica to another node from which the response is expected because the response is sent via the replication channel. In other words, there should be a full-mesh connection between the nodes.