Skip to content
This repository has been archived by the owner on Jan 13, 2025. It is now read-only.

filters crds values in parallel when responding to gossip pull-requests #18877

Merged
merged 1 commit into from
Jul 26, 2021

Conversation

behzadnouri
Copy link
Contributor

@behzadnouri behzadnouri commented Jul 23, 2021

Problem

When responding to gossip pull-requests, filter_crds_values takes a lot of time while holding onto read-lock:
https://github.com/solana-labs/solana/blob/f51d64868/gossip/src/crds_gossip_pull.rs#L509-L566

Summary of Changes

  • filter crds values in parallel.

@behzadnouri behzadnouri force-pushed the filter-crds-parallel branch 3 times, most recently from a90ead0 to 928f969 Compare July 24, 2021 16:51
@codecov
Copy link

codecov bot commented Jul 24, 2021

Codecov Report

Merging #18877 (8de8e2d) into master (f264511) will increase coverage by 0.0%.
The diff coverage is 93.2%.

@@           Coverage Diff           @@
##           master   #18877   +/-   ##
=======================================
  Coverage    82.8%    82.8%           
=======================================
  Files         444      444           
  Lines      126568   126582   +14     
=======================================
+ Hits       104838   104884   +46     
+ Misses      21730    21698   -32     

@behzadnouri behzadnouri force-pushed the filter-crds-parallel branch from 928f969 to 8de8e2d Compare July 24, 2021 18:41
@behzadnouri behzadnouri changed the title filters crds values in parallel filters crds values in parallel when responding to gossip pull-requests Jul 26, 2021
@behzadnouri
Copy link
Contributor Author

Left side is this code, versus current master code on the right.

cluster_info_stats2::handle_batch_pull_requests_time:
cluster_info_stats2 handle_batch_pull_requests_time

cluster_info_stats2::gossip_packets_dropped_count: Fewer packets are dropped because the responses are generated more quickly, and fewer packets are buffered:
https://github.com/solana-labs/solana/blob/f51d64868/gossip/src/cluster_info.rs#L2423-L2429
cluster_info_stats2 gossip_packets_dropped_count

@behzadnouri behzadnouri requested review from carllin and sakridge July 26, 2021 16:42
Copy link
Contributor

@sakridge sakridge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@behzadnouri behzadnouri merged commit f1198fc into solana-labs:master Jul 26, 2021
@behzadnouri behzadnouri deleted the filter-crds-parallel branch July 26, 2021 17:13
mergify bot pushed a commit that referenced this pull request Jul 26, 2021
…ts (#18877)

When responding to gossip pull-requests, filter_crds_values takes a lot of time
while holding onto read-lock:
https://github.com/solana-labs/solana/blob/f51d64868/gossip/src/crds_gossip_pull.rs#L509-L566

This commit will filter-crds-values in parallel using rayon thread-pools.

(cherry picked from commit f1198fc)

# Conflicts:
#	gossip/src/cluster_info.rs
#	gossip/src/crds_gossip_pull.rs
mergify bot added a commit that referenced this pull request Jul 26, 2021
…ts (backport #18877) (#18901)

* filters crds values in parallel when responding to gossip pull-requests (#18877)

When responding to gossip pull-requests, filter_crds_values takes a lot of time
while holding onto read-lock:
https://github.com/solana-labs/solana/blob/f51d64868/gossip/src/crds_gossip_pull.rs#L509-L566

This commit will filter-crds-values in parallel using rayon thread-pools.

(cherry picked from commit f1198fc)

# Conflicts:
#	gossip/src/cluster_info.rs
#	gossip/src/crds_gossip_pull.rs

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
@behzadnouri
Copy link
Contributor Author

TDS improvement with now 84% of stake upgrading to this code.

cluster_info_stats2::handle_batch_pull_requests_time:
cluster_info_stats2 handle_batch_pull_requests_time_tds

@brooksprumo brooksprumo mentioned this pull request Aug 23, 2021
mergify bot pushed a commit that referenced this pull request Aug 26, 2021
…ts (#18877)

When responding to gossip pull-requests, filter_crds_values takes a lot of time
while holding onto read-lock:
https://github.com/solana-labs/solana/blob/f51d64868/gossip/src/crds_gossip_pull.rs#L509-L566

This commit will filter-crds-values in parallel using rayon thread-pools.

(cherry picked from commit f1198fc)

# Conflicts:
#	core/src/cluster_info.rs
#	core/src/crds_gossip.rs
#	core/src/crds_gossip_pull.rs
#	core/src/gossip_service.rs
@behzadnouri behzadnouri removed the v1.6 label Aug 26, 2021
behzadnouri added a commit that referenced this pull request Sep 8, 2021
…ts (#18877)

When responding to gossip pull-requests, filter_crds_values takes a lot of time
while holding onto read-lock:
https://github.com/solana-labs/solana/blob/f51d64868/gossip/src/crds_gossip_pull.rs#L509-L566

This commit will filter-crds-values in parallel using rayon thread-pools.

(cherry picked from commit f1198fc)
mergify bot added a commit that referenced this pull request Sep 8, 2021
…ts (backport #18877) (#19450)

* filters crds values in parallel when responding to gossip pull-requests (#18877)

When responding to gossip pull-requests, filter_crds_values takes a lot of time
while holding onto read-lock:
https://github.com/solana-labs/solana/blob/f51d64868/gossip/src/crds_gossip_pull.rs#L509-L566

This commit will filter-crds-values in parallel using rayon thread-pools.

(cherry picked from commit f1198fc)

* removes backport merge conflicts

Co-authored-by: behzad nouri <behzadnouri@gmail.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants