Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(gossipsub): introduce backpressure #5595

Open
wants to merge 19 commits into
base: master
Choose a base branch
from
Open

feat(gossipsub): introduce backpressure #5595

wants to merge 19 commits into from
+2,186 −1,057

Conversation

jxs
Copy link
Member

@jxs jxs commented Sep 10, 2024

Description

superseeds #4914 with some changes and improvements, namely:

  • introduce a Delay for Forward and Publish messages, messages that take more than the configured delay to be sent are discarded
  • introduce scoring and penalize slow peers
  • remove control pool
  • report slow peers with the number of failed messages

@jxs jxs force-pushed the impl-gossipsub-backpressure-2 branch from 6d406e1 to 5e349f5 Compare September 10, 2024 22:44
@jxs jxs force-pushed the impl-gossipsub-backpressure-2 branch from 5e349f5 to 00cde64 Compare September 10, 2024 22:52
ackintosh
ackintosh reviewed Sep 15, 2024
View reviewed changes
Copy link
Contributor

@ackintosh ackintosh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’ve left a few small comments.

protocols/gossipsub/src/behaviour.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/peer_score.rs Outdated Show resolved Hide resolved
guillaumemichel
guillaumemichel reviewed Sep 26, 2024
View reviewed changes
Copy link
Contributor

@guillaumemichel guillaumemichel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be great to add new tests for the introduced backpressure

protocols/gossipsub/src/peer_score/params.rs Show resolved Hide resolved
protocols/gossipsub/src/config.rs Show resolved Hide resolved
protocols/gossipsub/src/config.rs Show resolved Hide resolved
protocols/gossipsub/src/config.rs Show resolved Hide resolved
protocols/gossipsub/src/types.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/types.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/types.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/types.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/peer_score.rs Show resolved Hide resolved
protocols/gossipsub/src/behaviour.rs Outdated Show resolved Hide resolved
jxs
jxs commented Oct 2, 2024
View reviewed changes
Copy link
Member Author

@jxs jxs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the reviews Akihito and Gui!

protocols/gossipsub/src/behaviour.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/peer_score.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/behaviour.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/config.rs Show resolved Hide resolved
protocols/gossipsub/src/types.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/types.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/types.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/types.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/types.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/types.rs Outdated Show resolved Hide resolved
@jxs jxs mentioned this pull request Oct 25, 2024
Merged
4 tasks
@jxs jxs force-pushed the impl-gossipsub-backpressure-2 branch from 8564d5e to 887fe07 Compare October 31, 2024 10:27
jxs
jxs commented Oct 31, 2024
View reviewed changes
Copy link
Member Author

@jxs jxs left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be great to add new tests for the introduced backpressure

yup for sure thanks for the suggestion Gui! Added some tests

@jxs jxs force-pushed the impl-gossipsub-backpressure-2 branch from ee51578 to ec0e0b4 Compare November 4, 2024 10:29
@jxs jxs force-pushed the impl-gossipsub-backpressure-2 branch from ec0e0b4 to c9e6602 Compare November 4, 2024 12:07
ackintosh
ackintosh approved these changes Nov 4, 2024
View reviewed changes
Copy link
Contributor

@ackintosh ackintosh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks João, LGTM!

elenaf9
elenaf9 reviewed Nov 18, 2024
View reviewed changes
Copy link
Contributor

@elenaf9 elenaf9 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few comments and nit picks.

protocols/gossipsub/src/behaviour.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/behaviour.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/types.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/types.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/types.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/handler.rs Show resolved Hide resolved
elenaf9
elenaf9 reviewed Nov 19, 2024
View reviewed changes
protocols/gossipsub/src/types.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/types.rs Show resolved Hide resolved
protocols/gossipsub/src/types.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/types.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/types.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/config.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/error.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/handler.rs Show resolved Hide resolved
protocols/gossipsub/src/types.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/behaviour.rs Outdated Show resolved Hide resolved
@elenaf9 elenaf9 mentioned this pull request Nov 20, 2024
Merged
4 tasks
@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 21, 2024

This pull request has merge conflicts. Could you please resolve them @jxs? 🙏

@jxs
Merge pull request #12 from elenaf9/gossipsub/sender_send_message
b626118 
refactor(gossipsub): use `send_message` for all RpcOut
@jxs jxs mentioned this pull request Nov 22, 2024
Merged
@elenaf9 elenaf9 mentioned this pull request Nov 22, 2024
Merged
4 tasks
jxs and others added 5 commits November 22, 2024 15:43
@jxs @elenaf9
Update protocols/gossipsub/src/behaviour.rs
c812627 
Co-authored-by: Elena Frank <elena.frank@proton.me>
@jxs
Merge pull request #13 from elenaf9/gossipsub/backpressure/readd-logs
1789386 
fix(gossipsub): readd lines accidently removed with #12
@jxs @elenaf9
Apply suggestions from code review
7665b2d 
Co-authored-by: Elena Frank <elena.frank@proton.me>
@jxs @elenaf9
Update protocols/gossipsub/src/types.rs
d795118 
Co-authored-by: Elena Frank <elena.frank@proton.me>
@jxs @elenaf9
Apply suggestions from code review
6fc1bb2 
Co-authored-by: Elena Frank <elena.frank@proton.me>
jxs
jxs commented Nov 22, 2024
View reviewed changes
Copy link
Member Author

@jxs jxs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think I addressed everything, can you take another look Elena? Thanks!

protocols/gossipsub/src/types.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/types.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/error.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/config.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/types.rs Outdated Show resolved Hide resolved
protocols/gossipsub/src/handler.rs Show resolved Hide resolved
protocols/gossipsub/src/behaviour.rs Show resolved Hide resolved
protocols/gossipsub/src/handler.rs Show resolved Hide resolved
@elenaf9
Copy link
Contributor

elenaf9 commented Nov 22, 2024 •
edited
Loading

I think the new rpc module was accidentally not included in 602c46f? 😇
But everything else looks good to me! Just one remaining open comment about whether in the case that the channel is full and publishing failed, we should also update the metrics same as when a message is dropped?

@jxs
Copy link
Member Author

jxs commented Nov 23, 2024

I think the new rpc module was accidentally not included in 602c46f? 😇 But everything else looks good to me! Just one remaining open comment about whether in the case that the channel is full and publishing failed, we should also update the metrics same as when a message is dropped?

ah right! Added the rpc module. The unresolved conversations were from before your PR was merged, now metrics are being record on send_message and cover the cases you commented right?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@elenaf9 elenaf9 elenaf9 left review comments

@AgeManning AgeManning AgeManning left review comments

@guillaumemichel guillaumemichel guillaumemichel approved these changes

@ackintosh ackintosh ackintosh approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@jxs @elenaf9 @ackintosh @AgeManning @guillaumemichel