Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decouple channel management from host session management #2801

Merged
merged 87 commits into from
Sep 28, 2021
Merged
Show file tree
Hide file tree
Changes from 66 commits
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
da2e00a
Collection of deletions
eddyashton Jun 24, 2021
a391755
Shuffle where things are constructed, WIP attempt nonce
eddyashton Jun 25, 2021
f8d2164
Move NodeToNode into NodeState too
eddyashton Jun 25, 2021
efcd67e
Move NodeToNodeImpl to NodeToNodeChannelManager
eddyashton Jun 25, 2021
da1a4a3
Collapse ChannelManager into N2NCM
eddyashton Jun 25, 2021
334ae9f
Trim API
eddyashton Jun 28, 2021
b6a80c8
Simplify API to host
eddyashton Jun 28, 2021
390354a
Bash tests to use simpler API
eddyashton Jun 28, 2021
b985aa8
Progress
eddyashton Jun 28, 2021
9c2f763
Progress
eddyashton Jun 28, 2021
070357b
Restore final test
eddyashton Jun 28, 2021
8ed4395
Restore interrupted key exchange test
eddyashton Jun 29, 2021
c073a53
WIP: Test robustness with junk
eddyashton Jun 29, 2021
e76dc93
Use pretty-printing state machine for ChannelState
eddyashton Jun 29, 2021
4132194
Add attempt nonce in all messages, to pass new test
eddyashton Jun 29, 2021
c6a4a05
Tidy
eddyashton Jun 29, 2021
b6af9fb
Response is resubmittable
eddyashton Jun 29, 2021
4136770
Shuffle functions
eddyashton Jun 29, 2021
6e3e205
Make some more state private
eddyashton Jun 29, 2021
2ddb25d
Consistent TRACE logging
eddyashton Jun 29, 2021
2df26af
Retain structure parsed in node_inbound
eddyashton Jun 29, 2021
c1ff63a
Avoid unnecessary copy off ringbuffer
eddyashton Jun 29, 2021
1a41bb1
Use normal ringbuffer def for kex node_outbound
eddyashton Jun 30, 2021
1a73e31
For normal sends too
eddyashton Jun 30, 2021
5ac53c6
Snark hunt, including too much logging
eddyashton Jun 30, 2021
f17fddc
Remove some unnecessary hacks/logging
eddyashton Jul 1, 2021
92bd641
Clean up after tests
eddyashton Jul 1, 2021
6e28450
Remove some TODOs that are already fixed
eddyashton Jul 1, 2021
9d44a0b
Remove more debug logging
eddyashton Jul 2, 2021
58597b2
Restore code accidentally removed from the _with_payload path
eddyashton Jul 2, 2021
8c74de7
Remove initiation nonce, simplify serialisation patterns
eddyashton Jul 5, 2021
81d86e3
Fix tests to handle this brave new world
eddyashton Jul 5, 2021
a4b7a4a
Logging tweaks
eddyashton Jul 5, 2021
8f83110
Remove resends
eddyashton Jul 5, 2021
4265eb5
Sometimes we need to re-initiate
eddyashton Jul 7, 2021
f6628d0
Remove unused, don't always double-init
eddyashton Jul 7, 2021
41114ab
Fix tests
eddyashton Jul 7, 2021
498b51c
Merge branch 'main' of github.com:microsoft/CCF into only_host_tracks…
eddyashton Jul 7, 2021
a449f81
Remove debug logging
eddyashton Jul 7, 2021
66e9694
Don't send protocol_version in FINAL message
eddyashton Jul 8, 2021
27f9ab7
Merge branch 'main' of github.com:microsoft/CCF into eda_channels_1
eddyashton Jul 8, 2021
e3f3bda
Merge branch 'main' of github.com:microsoft/CCF into eda_channels_1
eddyashton Jul 12, 2021
8f247d6
Format
eddyashton Jul 12, 2021
56c5131
Remove some unnecessary changes
eddyashton Jul 12, 2021
3be7e50
Update host-side connection management
eddyashton Jul 13, 2021
1885eb2
Right, don't close so aggressively on reset
eddyashton Jul 13, 2021
bd8fab9
Tweaks
eddyashton Jul 13, 2021
1ad79ab
Merge branch 'main' of github.com:microsoft/CCF into eda_channels_1
eddyashton Jul 13, 2021
a3f4871
Oops
eddyashton Jul 13, 2021
df6f462
Handle new message types
eddyashton Jul 14, 2021
1e5f69f
Merge branch 'main' of github.com:microsoft/CCF into eda_channels_1
eddyashton Jul 14, 2021
7b09022
Tweaks
eddyashton Jul 15, 2021
66dbdc5
Merge branch 'main' of github.com:microsoft/CCF into eda_channels_2
eddyashton Jul 15, 2021
03fb932
Trigger Daily
eddyashton Jul 15, 2021
91e0a23
Format
eddyashton Jul 15, 2021
2b97d85
Don't reconnect, just recreate connections
eddyashton Jul 15, 2021
35ad77e
D'oh!
eddyashton Jul 15, 2021
e8c50e5
Format
eddyashton Jul 16, 2021
0323f2c
Tentative Raft fix
eddyashton Jul 16, 2021
bf3335f
Merge branch 'main' of github.com:microsoft/CCF into eda_channels_2
eddyashton Jul 16, 2021
4d30540
Merge branch 'main' into eda_channels_2
achamayou Jul 21, 2021
3f22406
Revert tentative fix
eddyashton Jul 23, 2021
cf0dfa6
Merge branch 'main' of github.com:microsoft/CCF into eda_channels_2
eddyashton Jul 23, 2021
d8cb9ae
Merge branch 'eda_channels_2' of github.com:eddyashton/CCF into eda_c…
eddyashton Jul 23, 2021
f603adb
Farmot
eddyashton Jul 23, 2021
cbd5971
Merge branch 'main' into eda_channels_2
achamayou Jul 23, 2021
f013910
Merge branch 'main' of github.com:microsoft/CCF into eda_channels_2
eddyashton Jul 23, 2021
38e49cf
Merge branch 'main' into eda_channels_2
achamayou Jul 23, 2021
21f2eac
Merge branch 'main' of github.com:microsoft/CCF into eda_channels_2
eddyashton Aug 5, 2021
4f9f7bf
Merge branch 'main' into eda_channels_2
achamayou Aug 6, 2021
a34b17a
Merge branch 'main' into eda_channels_2
achamayou Aug 8, 2021
7efb2ec
Merge branch 'main' into eda_channels_2
achamayou Aug 10, 2021
a9ccd09
Merge branch 'main' of github.com:microsoft/CCF into eda_channels_2
eddyashton Sep 14, 2021
4b20c85
Merge fixups
eddyashton Sep 14, 2021
84e4b29
Merge branch 'eda_channels_2' of github.com:eddyashton/CCF into eda_c…
eddyashton Sep 15, 2021
2811b02
Oops
eddyashton Sep 15, 2021
946c2e3
Sure, abandon TODOs
eddyashton Sep 15, 2021
e3b5bf2
Add a small delay between initiate attempts, to be politer to old nodes
eddyashton Sep 22, 2021
544e6c4
Merge branch 'main' of github.com:microsoft/CCF into eda_channels_2
eddyashton Sep 22, 2021
0d9896a
Minor fixes
eddyashton Sep 22, 2021
2a5f251
Merge branch 'main' of github.com:microsoft/CCF into eda_channels_2
eddyashton Sep 22, 2021
8d628a1
Display iteration of primary isolation test
eddyashton Sep 27, 2021
428570a
Correct order of formatting arguments
eddyashton Sep 27, 2021
ccf48e6
" "
eddyashton Sep 27, 2021
8e7d35d
Make partitions symmetric: Block response traffic too
eddyashton Sep 27, 2021
29ccad6
Merge branch 'main' of github.com:microsoft/CCF into eda_channels_2
eddyashton Sep 27, 2021
2bda96d
Update comment
eddyashton Sep 27, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .daily_canary
Original file line number Diff line number Diff line change
@@ -1 +1 @@
cough
LET them be triggered
50 changes: 26 additions & 24 deletions src/consensus/aft/raft.h
Original file line number Diff line number Diff line change
Expand Up @@ -824,6 +824,7 @@ namespace aft
std::make_unique<NonceRevealCallback>(*this, from, std::move(r));
break;
}

case bft_view_change:
{
RequestViewChangeMsg r =
Expand Down Expand Up @@ -1295,10 +1296,12 @@ namespace aft
(state->new_view_idx > prev_idx) && (state->new_view_idx <= end_idx);

LOG_DEBUG_FMT(
"Send append entries from {} to {}: {} to {} ({})",
state->my_node_id.trim(),
to.trim(),
start_idx,
"Send append entries from {} to {}: ({}.{}, {}.{}] ({})",
state->my_node_id,
to,
prev_term,
prev_idx,
term_of_idx,
end_idx,
state->commit_idx);

Expand Down Expand Up @@ -1480,6 +1483,7 @@ namespace aft
prev_term,
r.prev_term);
}

send_append_entries_response(from, AppendEntriesResponseType::FAIL);
return;
}
Expand Down Expand Up @@ -2123,12 +2127,10 @@ namespace aft
void send_append_entries_response(
ccf::NodeId to, AppendEntriesResponseType answer)
{
LOG_DEBUG_FMT(
"Send append entries response from {} to {} for index {}: {}",
state->my_node_id.trim(),
to.trim(),
state->last_idx,
answer);
if (answer == AppendEntriesResponseType::REQUIRE_EVIDENCE)
{
state->requested_evidence_from = to;
}

if (answer == AppendEntriesResponseType::REQUIRE_EVIDENCE)
{
Expand All @@ -2148,6 +2150,13 @@ namespace aft
state->commit_idx :
state->last_idx;

LOG_DEBUG_FMT(
"Send append entries response from {} to {} for index {}: {}",
state->my_node_id.trim(),
to.trim(),
matching_idx,
answer);

AppendEntriesResponse response = {{raft_append_entries_response},
state->current_view,
matching_idx,
Expand Down Expand Up @@ -2696,10 +2705,6 @@ namespace aft
{
for (auto it = nodes.begin(); it != nodes.end(); ++it)
{
channels->create_channel(
it->first,
it->second.node_info.hostname,
it->second.node_info.port);
send_request_vote(it->first);
}
}
Expand Down Expand Up @@ -3283,22 +3288,19 @@ namespace aft
auto index = state->last_idx + 1;
nodes.try_emplace(node_info.first, node_info.second, index, 0);

if (
replica_state == kv::ReplicaState::Leader ||
consensus_type == ConsensusType::BFT)
{
channels->create_channel(
node_info.first,
node_info.second.hostname,
node_info.second.port);
}
channels->associate_node_address(
node_info.first, node_info.second.hostname, node_info.second.port);

if (replica_state == kv::ReplicaState::Leader)
{
send_append_entries(node_info.first, index);
}

LOG_INFO_FMT("Added raft node {}", node_info.first);
LOG_INFO_FMT(
"Added raft node {} ({}:{})",
node_info.first,
node_info.second.hostname,
node_info.second.port);
}
}
}
Expand Down
17 changes: 7 additions & 10 deletions src/consensus/aft/test/logging_stub.h
Original file line number Diff line number Diff line change
Expand Up @@ -115,18 +115,13 @@ namespace aft
return std::nullopt;
}

void create_channel(
void associate_node_address(
const ccf::NodeId& peer_id,
const std::string& peer_hostname,
const std::string& peer_service,
size_t message_limit = ccf::Channel::default_message_limit) override
const std::string& peer_service) override
{}

void destroy_channel(const ccf::NodeId& peer_id) override {}

void destroy_all_channels() override {}

void close_all_outgoing() override {}
void close_channel(const ccf::NodeId& peer_id) override {}

bool send_authenticated(
const ccf::NodeId& to,
Expand All @@ -148,9 +143,11 @@ namespace aft
return true;
}

void recv_message(
bool recv_channel_message(
const ccf::NodeId& from, const uint8_t* data, size_t size) override
{}
{
return true;
}

void initialize(
const ccf::NodeId& self_id,
Expand Down
15 changes: 10 additions & 5 deletions src/crypto/symmetric_key.h
Original file line number Diff line number Diff line change
Expand Up @@ -96,14 +96,19 @@ namespace crypto
return serial_hdr;
}

void deserialise(const std::vector<uint8_t>& serial_hdr)
void deserialise(const std::vector<uint8_t>& ser)
{
auto data_ = serial_hdr.data();
auto size = serial_hdr.size();
auto data = ser.data();
auto size = ser.size();

deserialise(data, size);
}

void deserialise(const uint8_t*& data, size_t& size)
{
memcpy(
tag, serialized::read(data_, size, GCM_SIZE_TAG).data(), GCM_SIZE_TAG);
memcpy(iv, serialized::read(data_, size, SIZE_IV).data(), SIZE_IV);
tag, serialized::read(data, size, GCM_SIZE_TAG).data(), GCM_SIZE_TAG);
memcpy(iv, serialized::read(data, size, SIZE_IV).data(), SIZE_IV);
}
};

Expand Down
3 changes: 1 addition & 2 deletions src/host/main.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -645,8 +645,7 @@ int main(int argc, char** argv)
// This includes DNS resolution and potentially dynamic port assignment (if
// requesting port 0). The hostname and port may be modified - after calling
// it holds the final assigned values.
asynchost::NodeConnectionsTickingReconnect node(
20ms, //< Flush reconnections every 20ms
asynchost::NodeConnections node(
bp.get_dispatcher(),
ledger,
writer_factory,
Expand Down
Loading