Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reinstate the use of bootstrap DoS mitigations removed along with tokio #3821

Merged
merged 93 commits into from
Apr 24, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
93 commits
Select commit Hold shift + click to select a range
7ae5b2d
Make establisher `get_listener` non-async
Ben-PH Feb 27, 2023
bdd8e54
Make establisher `fn get_connector` non-async
Ben-PH Feb 27, 2023
b77875c
Set the mockers to use a TcpStream as a duplex
Ben-PH Feb 27, 2023
4e6c89f
fix(bootstrap) : fix base_bootstrap_ip
Feb 28, 2023
bf0301c
Duplex -> tokio::net::TcpStream. same for Listener
Ben-PH Mar 3, 2023
baac844
unify listener interface through trait
Ben-PH Mar 3, 2023
1e00ce4
remove `#[cfg(test)]` from bootstrap establisher
Ben-PH Mar 3, 2023
77d0b8e
Add FIXME to where static lifetime is being used as a hot-fix
Ben-PH Mar 3, 2023
710a8c8
Light house-keeping
Ben-PH Mar 3, 2023
4d14a09
Clear clippy lint
Ben-PH Mar 3, 2023
7c131db
Some code-comments
Ben-PH Mar 3, 2023
2150b03
Using a blocking tcp interface, but things lock up
Ben-PH Mar 13, 2023
2e086c1
Fix blocking by making the tokio conversion using non-blocking objects
Ben-PH Mar 14, 2023
a637e00
Remove the async Tcp stuff from mock connector
Ben-PH Mar 14, 2023
725902f
Use less async channels. Use Arc<AtomicBool> as signaller
Ben-PH Mar 14, 2023
0236535
Fewer async channels
Ben-PH Mar 14, 2023
b32246b
So close to not using tokio...
Ben-PH Mar 14, 2023
a7f3c51
accept method now synchronous!!!
Ben-PH Mar 14, 2023
c1c0936
Merge remote-tracking branch 'origin/testnet_21' into bootstrap/sync-…
Ben-PH Mar 15, 2023
e8d53f1
make connect synchronous
Ben-PH Mar 15, 2023
2cc4256
Some cleanup
Ben-PH Mar 15, 2023
fcdf28e
Clear clippy lint
Ben-PH Mar 15, 2023
5732529
Fix upstream merge error
Ben-PH Mar 15, 2023
bf7f89f
more cleanup
Ben-PH Mar 15, 2023
78459e0
Make listener blocking
Ben-PH Mar 22, 2023
dffa060
Make the Duplex a trait, thus mockable
Ben-PH Mar 24, 2023
ea51448
Start the server with the listener instead of estab trait
Ben-PH Mar 27, 2023
436517b
Update comments
Ben-PH Mar 27, 2023
11a3b24
Remove mock-establisher interface entirely
Ben-PH Mar 27, 2023
e0a3089
Move the default get-listener into a struct method
Ben-PH Mar 27, 2023
17783ad
Give mock setup variables clearer names
Ben-PH Mar 27, 2023
660e7e2
Add mocking todos
Ben-PH Mar 27, 2023
4802558
Remove `dbg!`s
Ben-PH Mar 29, 2023
e990892
Self review
Ben-PH Mar 29, 2023
c53a842
Make the network command sender mockable
Ben-PH Mar 27, 2023
40e8f60
Use auto-mock to get the bootstrap peers
Ben-PH Mar 27, 2023
e224356
Mock the consensus controller implementation when testing bootstrap
Ben-PH Mar 27, 2023
ed056e8
Replace the mock establisher with `automock` of listener and connector
Ben-PH Mar 29, 2023
877fb00
Clarify the mock-connection setup
Ben-PH Mar 29, 2023
512a4ff
Remove the vestigial mock_establisher.rs file
Ben-PH Mar 29, 2023
337d3fd
Move some of `get_stat` pre-loop code to helper functions
Ben-PH Mar 29, 2023
4bc6611
Move everything from tokio to std::net::TcpStream
Ben-PH Mar 29, 2023
c9f107d
Comment out the use of tokio timeeouts
Ben-PH Mar 29, 2023
210e52e
Propogate errors instead of unwrapping
Ben-PH Mar 29, 2023
840cfa5
Don't panic if you don't have a bootstrap listen address
Ben-PH Mar 29, 2023
dcf2351
Merge remote-tracking branch 'origin/testnet_21' into bootstrap/sync-…
Ben-PH Mar 29, 2023
0721d10
Remove vestigial `DefaultEstablisher`
Ben-PH Mar 29, 2023
04f30e7
Merge branch 'bootstrap/sync-accept' into bootstrap/mocked-and-sync
Ben-PH Mar 29, 2023
f128231
Refactor the connector
Ben-PH Mar 29, 2023
36d6101
General Cleanup
Ben-PH Mar 29, 2023
7cc9e45
De-async more methods and start working on `next` blockage
Ben-PH Mar 30, 2023
aa38179
Fix clippy and compilation warn/error
Ben-PH Mar 30, 2023
757ac3c
Make the boostrap test sync
Ben-PH Mar 30, 2023
6282b68
Integrate timeouts into sending and receiving messages
Ben-PH Mar 30, 2023
af20c1e
Fully clear out async in the bootstrap test
Ben-PH Mar 30, 2023
2bbd6ac
Get bootstrap test passing!
Ben-PH Mar 30, 2023
3b73bda
Make message sending from server sync
Ben-PH Mar 30, 2023
7c5c3d6
Get binder tests working
Ben-PH Mar 30, 2023
54954a7
Merge branch 'bootstrap/sync-accept' into bootstrap/mocked-and-sync
Ben-PH Mar 30, 2023
38d5fc2
Fix issues created by merge
Ben-PH Mar 30, 2023
22c5d96
Merge remote-tracking branch 'origin/testnet_21' into bootstrap/mocke…
Ben-PH Mar 30, 2023
a8e5d57
clear clippy lints
Ben-PH Mar 30, 2023
b1a4e35
Remove unused dependency
Ben-PH Mar 30, 2023
68c1efe
Remove `dbg!`s
Ben-PH Mar 30, 2023
cd8392c
Add timed-out variant to bootstrap error
Ben-PH Mar 31, 2023
f706b85
remove clippy::result_large_err (#3751)
modship Apr 3, 2023
c674533
remove #[allow(clippy::result_large_err)] in white_black_list.rs
Apr 3, 2023
45e37f6
why unwrap, when you have ?... ?
Ben-PH Apr 3, 2023
eb46446
improve error and readability (#3757)
modship Apr 4, 2023
3b299cf
Remove the Duplex trait.
Ben-PH Apr 4, 2023
d87fccc
Merge remote-tracking branch 'origin/testnet_21' into bootstrap/mocke…
Ben-PH Apr 4, 2023
805ce36
Fix Duplex deletion commit
Ben-PH Apr 4, 2023
5794404
Merge branch 'testnet_21' into bootstrap/mocked-and-sync
Ben-PH Apr 5, 2023
15f4496
Merge remote-tracking branch 'origin/testnet_21' into bootstrap/mocke…
Ben-PH Apr 6, 2023
8cc4f62
Merge remote-tracking branch 'origin/testnet_22' into bootstrap/mocke…
Ben-PH Apr 7, 2023
62e1781
Removes the `tokio` dependency from the bootstrap module (#3782)
Ben-PH Apr 7, 2023
d92ce87
Revert "Removes the `tokio` dependency from the bootstrap module (#37…
Ben-PH Apr 7, 2023
81c1f41
Apply self-review changes
Ben-PH Apr 11, 2023
5bb53af
Hide mocks behind testing flags/builds
Ben-PH Apr 11, 2023
3ec51db
Clear cargo-doc warnings
Ben-PH Apr 11, 2023
0aa8086
More whack-a-mole with CI ~warns~ errors
Ben-PH Apr 11, 2023
f35e902
Remove the dependency on tokio in the bootstrap module (#3796)
Ben-PH Apr 12, 2023
f7583f2
Fix missing network-controller export of runtime constructor
Ben-PH Apr 12, 2023
de44109
Comment providing context of why an unused variable is retained
Ben-PH Apr 12, 2023
caf4d87
Minor cleanup of mocking code
Ben-PH Apr 12, 2023
56122d1
Bootstrap server binding read-message in a single `read_exact` (#3774)
Ben-PH Apr 12, 2023
4bea0ab
Expand the bandwith limit todo-comment
Ben-PH Apr 12, 2023
748f7ac
Fix dependency management for when testing bootstrap server directly
Ben-PH Apr 12, 2023
2a3b0f4
Bootstrap/peek streams (#3820)
Ben-PH Apr 12, 2023
72d1f6b
Set timeouts according to the time-window alloted
Ben-PH Apr 12, 2023
a6e9653
Address self-reveiw comments
Ben-PH Apr 12, 2023
e0e80e2
Clear out resolved todo
Ben-PH Apr 12, 2023
c49d6c5
Merge branch 'testnet_22' into bootstrap/time-trial-revival
Ben-PH Apr 24, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions massa-bootstrap/src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,8 @@ pub enum BootstrapError {
BlackListed(String),
/// IP {0} is not in the whitelist
WhiteListed(String),
/// The bootstrap process ended prematurely - e.g. too much time elapsed
Interupted(String),
}

/// # Platform-specific behavior
Expand Down
65 changes: 53 additions & 12 deletions massa-bootstrap/src/server.rs
Original file line number Diff line number Diff line change
Expand Up @@ -471,7 +471,7 @@ fn run_bootstrap_session(
deadline,
mip_store,
);
// TODO: handle the deadline management

// This drop allows the server to accept new connections before having to complete the error notifications
// account for this session being finished, as well as the root-instance
massa_trace!("bootstrap.session.finished", {
Expand Down Expand Up @@ -517,6 +517,7 @@ pub fn stream_bootstrap_information(
mut last_ops_step: StreamingStep<Slot>,
mut last_consensus_step: StreamingStep<PreHashSet<BlockId>>,
mut send_last_start_period: bool,
bs_deadline: &Instant,
write_timeout: Duration,
) -> Result<(), BootstrapError> {
loop {
Expand Down Expand Up @@ -657,6 +658,7 @@ pub fn stream_bootstrap_information(
}

// If the consensus streaming is finished (also meaning that consensus slot == final state slot) exit
// We don't bother with the bs-deadline, as this is the last step of the bootstrap process - defer to general write-timeout
if final_state_global_step.finished()
&& final_state_changes_step.finished()
&& last_consensus_step.finished()
Expand All @@ -665,6 +667,9 @@ pub fn stream_bootstrap_information(
break;
}

let Some(write_timeout) = step_timeout_duration(bs_deadline, &write_timeout) else {
return Err(BootstrapError::Interupted("insufficient time left to provide next bootstrap part".to_string()));
};
// At this point we know that consensus, final state or both are not finished
server.send_msg(
write_timeout,
Expand All @@ -685,6 +690,17 @@ pub fn stream_bootstrap_information(
Ok(())
}

// derives the duration allowed for a step in the bootstrap process.
// Returns None if the deadline for the entire bs-process has been reached
fn step_timeout_duration(bs_deadline: &Instant, step_timeout: &Duration) -> Option<Duration> {
let now = Instant::now();
if now >= *bs_deadline {
return None;
}

let remaining = *bs_deadline - now;
Some(std::cmp::min(remaining, *step_timeout))
}
#[allow(clippy::too_many_arguments)]
fn manage_bootstrap(
bootstrap_config: &BootstrapConfig,
Expand All @@ -693,15 +709,27 @@ fn manage_bootstrap(
version: Version,
consensus_controller: Box<dyn ConsensusController>,
network_command_sender: NetworkCommandSender,
_deadline: Instant,
deadline: Instant,
mip_store: MipStore,
) -> Result<(), BootstrapError> {
massa_trace!("bootstrap.lib.manage_bootstrap", {});
let read_error_timeout: Duration = bootstrap_config.read_error_timeout.into();

// TODO: make network_command_sender non-async and remove both the variable and the method
let rt_hack = massa_network_exports::make_runtime();

server.handshake_timeout(version, Some(bootstrap_config.read_timeout.into()))?;
let Some(hs_timeout) = step_timeout_duration(&deadline, &bootstrap_config.read_timeout.to_duration()) else {
return Err(BootstrapError::Interupted("insufficient time left to begin handshake".to_string()));
};

server.handshake_timeout(version, Some(hs_timeout))?;

// Check for error from client
if Instant::now() + read_error_timeout >= deadline {
return Err(BootstrapError::Interupted(
"insufficient time to check for error from client".to_string(),
));
};
match server.next_timeout(Some(read_error_timeout)) {
Err(BootstrapError::TimedOut(_)) => {}
Err(e) => return Err(e),
Expand All @@ -711,25 +739,33 @@ fn manage_bootstrap(
Ok(msg) => return Err(BootstrapError::UnexpectedClientMessage(Box::new(msg))),
};

let write_timeout: Duration = bootstrap_config.write_timeout.into();

// Sync clocks.
let server_time = MassaTime::now()?;

// Sync clocks
let send_time_timeout =
step_timeout_duration(&deadline, &bootstrap_config.write_timeout.to_duration());
let Some(next_step_timeout) = send_time_timeout else {
return Err(BootstrapError::Interupted("insufficient time left to send server time".to_string()));
};
server.send_msg(
write_timeout,
next_step_timeout,
BootstrapServerMessage::BootstrapTime {
server_time,
server_time: MassaTime::now()?,
version,
},
)?;

loop {
match server.next_timeout(Some(bootstrap_config.read_timeout.into())) {
let Some(read_timeout) = step_timeout_duration(&deadline, &bootstrap_config.read_timeout.to_duration()) else {
return Err(BootstrapError::Interupted("insufficient time left to process next message".to_string()));
};
match server.next_timeout(Some(read_timeout)) {
Err(BootstrapError::TimedOut(_)) => break Ok(()),
Err(e) => break Err(e),
Ok(msg) => match msg {
BootstrapClientMessage::AskBootstrapPeers => {
let Some(write_timeout) = step_timeout_duration(&deadline, &bootstrap_config.write_timeout.to_duration()) else {
return Err(BootstrapError::Interupted("insufficient time left to respond te request for peers".to_string()));
};

server.send_msg(
write_timeout,
BootstrapServerMessage::BootstrapPeers {
Expand Down Expand Up @@ -760,11 +796,16 @@ fn manage_bootstrap(
last_ops_step,
last_consensus_step,
send_last_start_period,
write_timeout,
&deadline,
bootstrap_config.write_timeout.to_duration(),
)?;
}
BootstrapClientMessage::AskBootstrapMipStore => {
let vs = mip_store.0.read().to_owned();
let Some(write_timeout) = step_timeout_duration(&deadline, &bootstrap_config.write_timeout.to_duration()) else {
return Err(BootstrapError::Interupted("insufficient time left to respond te request for mip-store".to_string()));
};

server.send_msg(
write_timeout,
BootstrapServerMessage::BootstrapMipStore { store: vs.clone() },
Expand Down