Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Security: Avoid reconnecting to peers that are likely unreachable #3030

Merged

Conversation

jvff
Copy link
Contributor

@jvff jvff commented Nov 5, 2021

Motivation

Zebra would keep trying to connect to Failed peers, even if they are likely unreachable and will always fail. This is a Denial-of-Service risk and places extra load on the network.

This closes #1865.

Solution

Filter the list of peers to reconnect to so that it doesn't contain peers that are likely unreachable.

A peer is considered to be probably unreachable if it was last seen (or was reported last seen) more than three days ago and if the previous connection attempt to it failed.

Review

I had discussed the implementation solution with @teor2345.

Reviewer Checklist

  • Code implements Specs and Designs
  • Tests for Expected Behaviour
  • Tests for Errors

Follow Up Work

Peers that are likely to be unreachable should be removed from the AddressBook. This might be part of #1873.

Make it simpler to construct a `Duration32` representing a certain
number of days.
@jvff jvff added P-High C-security Category: Security issues labels Nov 5, 2021
@jvff jvff requested a review from teor2345 November 5, 2021 22:20
@jvff jvff self-assigned this Nov 5, 2021
@jvff jvff force-pushed the avoid-reconnecting-to-unreachable-peers branch from d037afc to 646a6a8 Compare November 5, 2021 22:23
jvff added 3 commits November 5, 2021 22:47
A helper method to check if a peer was never seen before or if it was
last seen a long time ago. This will be one of the conditions to
consider a peer as unreachable.
A helper method to check if a peer should be considered unreachable. It
is considered unreachable if recent connection attempts have failed and
it was not recently seen.

If a peer is considered unreachable, Zebra shouldn't attempt to connect
to it again.
A peer is probably unreachable if it was last seen a long time ago and
if it's last connection attempt failed.
@jvff jvff force-pushed the avoid-reconnecting-to-unreachable-peers branch from 646a6a8 to 3905785 Compare November 5, 2021 22:47
jvff added 3 commits November 6, 2021 13:22
Redo the calculation on arbitrary `MetaAddr`s.
Redo the calculation on arbitrary `MetaAddr`s.
Given an `AddressBook` with a list of arbitrary `MetaAddr`s, check that
none of the peers listed for a reconnection is probably unreachable.
@jvff jvff force-pushed the avoid-reconnecting-to-unreachable-peers branch from 3905785 to 0b3fc8d Compare November 6, 2021 13:22
Copy link
Contributor

@teor2345 teor2345 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, I just want to double-check an edge case.

I also have some name suggestions that might make the code easier to read.

(Sorry, I merged the latest main branch into this PR, must have got confused with another one.)

zebra-network/src/meta_addr.rs Outdated Show resolved Hide resolved
zebra-network/src/address_book.rs Outdated Show resolved Hide resolved
zebra-network/src/address_book/tests/prop.rs Outdated Show resolved Hide resolved
zebra-network/src/constants.rs Outdated Show resolved Hide resolved
zebra-network/src/address_book.rs Outdated Show resolved Hide resolved
zebra-network/src/meta_addr.rs Outdated Show resolved Hide resolved
Copy link
Contributor

@teor2345 teor2345 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to test this code in an integration test?
What happens if Zebra has no recent peers?

zebra-network/src/address_book.rs Outdated Show resolved Hide resolved
jvff and others added 5 commits November 8, 2021 18:50
Remove the double negative from the name.

Co-authored-by: teor <teor@riseup.net>
Make the purpose of the constant clearer.

Co-authored-by: teor <teor@riseup.net>
Remove the double negative from the name.
Avoid having to negate the result of the method in security critical
filter.
Make sure the check is used in any place that requires a peer that's
ready for a connection attempt.
@jvff jvff force-pushed the avoid-reconnecting-to-unreachable-peers branch from 52f0076 to 01d63d1 Compare November 8, 2021 20:49
@jvff
Copy link
Contributor Author

jvff commented Nov 8, 2021

Do we need to test this code in an integration test? What happens if Zebra has no recent peers?

What do you mean by an integration test? Something like the acceptance tests we have?

Copy link
Contributor

@teor2345 teor2345 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some optional comment tweaks.

zebra-network/src/address_book.rs Outdated Show resolved Hide resolved
zebra-network/src/address_book/tests/prop.rs Outdated Show resolved Hide resolved
zebra-network/src/meta_addr.rs Outdated Show resolved Hide resolved
@teor2345
Copy link
Contributor

teor2345 commented Nov 8, 2021

Do we need to test this code in an integration test? What happens if Zebra has no recent peers?

What do you mean by an integration test? Something like the acceptance tests we have?

I was thinking of an integration test for all of zebra_network::init. I don't think we need to test zebrad, because it doesn't modify the address book.

Here's a similar test:

/// Open a local listener on `listen_addr` for `network`.
/// Asserts that the local listener address works as expected.
async fn local_listener_port_with(listen_addr: SocketAddr, network: Network) {
let config = Config {
listen_addr,
network,
// Stop Zebra making outbound connections
initial_mainnet_peers: HashSet::new(),
initial_testnet_peers: HashSet::new(),
..Config::default()
};

But you'd want to stuff the active address book with outdated peers.

Here's one way to generate fake peers:

// Create a list of dummy IPs, and initialize a config using them as the
// initial peers. The amount of these peers will overflow
// `PEERSET_INITIAL_TARGET_SIZE`.
let mut peers = HashSet::new();
for address_number in 0..PEER_COUNT {
peers.insert(
SocketAddr::new(Ipv4Addr::new(127, 1, 1, address_number as _).into(), 1).to_string(),
);
}

@mpguerra mpguerra requested a review from oxarbitrage November 9, 2021 15:04
@teor2345
Copy link
Contributor

teor2345 commented Nov 9, 2021

I was thinking of an integration test for all of zebra_network::init. I don't think we need to test zebrad, because it doesn't modify the address book.

Thinking about this a bit more...

The tricky part of this test is working out what the candidate set is doing, or if the network stack has hung. So maybe we'll need to test the crawler task separately?

There are tests for the crawler task in the same module.

jvff and others added 2 commits November 10, 2021 18:35
Describe the goal of the test better.

Co-authored-by: teor <teor@riseup.net>
List the conditions as bullet points.

Co-authored-by: teor <teor@riseup.net>
@jvff
Copy link
Contributor Author

jvff commented Nov 10, 2021

The tricky part of this test is working out what the candidate set is doing, or if the network stack has hung. So maybe we'll need to test the crawler task separately?

This seems more complicated than anticipated. Maybe we should do this in a separate PR? I opened #3048 to do it later.

@teor2345
Copy link
Contributor

This seems more complicated than anticipated. Maybe we should do this in a separate PR? I opened #3048 to do it later.

Fair enough. I don't think these code changes are particularly risky by themselves, and we do have decent test coverage of the crawler from other tests in that same module.

I added two specific scenarios that I am concerned about to ticket #3048.

I'll try to be clearer about specific test scenarios next time. Even if we decide to do them in a later ticket, we need to be specific about what we're testing. Otherwise we won't know how to estimate or schedule the ticket.

Copy link
Contributor

@teor2345 teor2345 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's document what happens when peers have no last seen time.

zebra-network/src/meta_addr.rs Show resolved Hide resolved
zebra-network/src/meta_addr.rs Outdated Show resolved Hide resolved
@teor2345
Copy link
Contributor

I tried a manual merge, let's see if it works.

teor2345
teor2345 previously approved these changes Nov 10, 2021
@teor2345 teor2345 enabled auto-merge (squash) November 10, 2021 22:23
@teor2345 teor2345 merged commit 11b5a33 into ZcashFoundation:main Nov 10, 2021
@jvff jvff deleted the avoid-reconnecting-to-unreachable-peers branch November 23, 2021 22:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-security Category: Security issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Security: Running Zebra nodes should eventually stop trying to contact peers that always fail
2 participants