Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Switch to futures-0.3 channels #6283

Merged
merged 7 commits into from
Feb 7, 2021
Merged

fix: Switch to futures-0.3 channels #6283

merged 7 commits into from
Feb 7, 2021

Conversation

ktff
Copy link
Contributor

@ktff ktff commented Jan 29, 2021

Closes #6043

Replaces tokio-0.2 channels introduced in #5868 with futures-0.3 channels.

Todo

OpenQuestions

  • Should we also replace, or at least try to replace guided by benchmarks, other usages of tokio channels with future3 channels? (EDIT: Nope, better to leave that for a future issue)

Signed-off-by: ktf <krunotf@gmail.com>
@ktff ktff added the type: task Generic non-code related tasks label Jan 29, 2021
@ktff ktff requested review from a team and lukesteensen and removed request for a team January 29, 2021 15:03
@ktff ktff self-assigned this Jan 29, 2021
@ktff ktff marked this pull request as draft January 29, 2021 16:28
@lukesteensen
Copy link
Member

Should we also replace, or at least try to replace guided by benchmarks, other usages of tokio channels with future3 channels?

I think this can be driven by benchmarks and profiling. I would not go ahead and spend the time to change everything everywhere without good evidence that performance improves meaningfully.

Copy link
Member

@lukesteensen lukesteensen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So far this looks reasonable to me!

@ktff
Copy link
Contributor Author

ktff commented Jan 31, 2021

I would not go ahead and spend the time to change everything everywhere without good evidence that performance improves meaningfully.

I agree, then we'll postpone that to a different issue where it will be easier to bench the difference.

Signed-off-by: ktf <krunotf@gmail.com>
@@ -81,6 +81,7 @@ where
Box::pin(
input_rx
.map(Message::Process)
.fuse()
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixes a bug where if no events are sent before shutting down this, then into_future will poll this and get None which it will pass with the rest of the stream which will be polled again later therefor breaking the stream contract of not calling it after first None.

@jszwedko
Copy link
Member

jszwedko commented Feb 1, 2021

I was able to run the test harness against these changes and the latest nightly.

Specifically for the real world performance test mentioned in #6043 it looks like this is an improvement:

--------------------------------------------------------------------------------
Test Comparison
Test name: real_world_1_performance
Test configuration: default
Subject: vector
Versions: dev-56a30078b6f5c29f61650d9733c0873f35c749a2 nightly/2021-02-01
--------------------------------------------------------------------------------
Metric          | dev-56a30078b6f5c29f61650d9... | nightly/2021-02-01
----------------|--------------------------------|-------------------
Test count      | 1                              | 2                 
Duration (avg)  | 61s                            | 61.5s             
Duration (max)  | 61s                            | 62s               
CPU sys (max)   | 1.5 W                          | 1.8 (+16%)        
CPU usr (max)   | 100 (+0%)                      | 99.5 W            
Load 1m (avg)   | 1.8 (+19%)                     | 1.5 W             
Mem used (max)  | 325.3 MiB (+3%)                | 315.3 MiB W       
Disk read (avg) | 434.6 kib/s (-2%)              | 446.4 kib/s W     
Disk read (sum) | 25.9 MiB W                     | 26.9 MiB (+3%)    
Disk writ (sum) | 277.6 MiB (+554%)              | 42.4 MiB W        
Net recv (avg)  | 6.7 MiB/s W                    | 6.3 MiB/s (-5%)   
Net recv (sum)  | 407.6 MiB W                    | 389.7 MiB (-4%)   
Net send (sum)  | 261.5 MiB                      | 248.1 MiB         
TCP estab (avg) | 603                            | 597               
TCP syn (avg)   | 0                              | 0                 
TCP close (avg) | 0                              | 0                 
--------------------------------------------------------------------------------
W = winner
vector = dev-56a30078b6f5c29f61650d9733c0873f35c749a2
vector = nightly/2021-02-01

However I do see some less favorable results for other configurations, for example:

--------------------------------------------------------------------------------
Test Comparison
Test name: real_world_1_performance
Test configuration: high_concurrency
Subject: vector
Versions: dev-56a30078b6f5c29f61650d9733c0873f35c749a2 nightly/2021-02-01
--------------------------------------------------------------------------------
Metric          | dev-56a30078b6f5c29f61650d9... | nightly/2021-02-01
----------------|--------------------------------|-------------------
Test count      | 5                              | 2                 
Duration (avg)  | 63.2s                          | 62s               
Duration (max)  | 66s                            | 62s               
CPU sys (max)   | 2.4 W                          | 2.6 (+5%)         
CPU usr (max)   | 28.5 W                         | 32.8 (+15%)       
Load 1m (avg)   | 4.2 W                          | 4.7 (+13%)        
Mem used (max)  | 687.8 MiB (+5%)                | 654.2 MiB W       
Disk read (avg) | 879.7 kib/s W                  | 681.2 kib/s (-22%)
Disk read (sum) | 54 MiB (+31%)                  | 41.2 MiB W        
Disk writ (sum) | 114.2 MiB (+29%)               | 88.1 MiB W        
Net recv (avg)  | 13.9 MiB/s (-22%)              | 18 MiB/s W        
Net recv (sum)  | 876.4 MiB (-21%)               | 1.1 gib W         
Net send (sum)  | 566.4 MiB                      | 722.1 MiB         
TCP estab (avg) | 607.4                          | 602.5             
TCP syn (avg)   | 0                              | 8.5               
TCP close (avg) | 0                              | 0                 
--------------------------------------------------------------------------------
W = winner
vector = dev-56a30078b6f5c29f61650d9733c0873f35c749a2
vector = nightly/2021-02-01

My confidence in the test harness is lower than it has been though, for this magnitude of difference. I think #6274 will help here.

Full test harness results: https://gist.github.com/jszwedko/9f4236899a0df1450d698f08d86a20bb

I'll try running the criterion benches.

@jszwedko
Copy link
Member

jszwedko commented Feb 1, 2021

I keep the criterion ones are already in CI 😄

For the topology benches I see:

2021-01-29T18:03:48.8351489Z pipe/pipe_simple                                                                                     38.998 ms  -3.2001%     24.455 MiB/s    +3.3059%           0.00  none
2021-01-29T18:03:48.8352485Z pipe/pipe_small_lines                                                                                31.661 ms  -1.6706%     308.45 KiB/s    +1.6989%           0.00  none
2021-01-29T18:03:48.8353287Z pipe/pipe_big_lines                                                                                  156.54 ms  -3.6279%     121.85 MiB/s    +3.7645%           0.00  none
2021-01-29T18:03:48.8354086Z pipe/pipe_multiple_writers                                                                           41.732 ms  -4.7469%     2.2852 MiB/s    +4.9835%           0.00  none
2021-01-29T18:03:48.8354975Z interconnected/interconnected                                                                        64.958 ms  -8.4663%     29.363 MiB/s    +9.2494%           0.02  none
2021-01-29T18:03:48.8355858Z transforms/transforms                                                                                77.984 ms  -1.4718%     13.452 MiB/s    +1.4938%           0.00  none
2021-01-29T18:03:48.8356683Z complex/complex                                                                                      3.0303 s   -5.6531%     unknown         unknown            0.00  none

Which does show some improvement, though not more than our noise threshold for those tests (20%). The fact that the improvement is consistent across all of them does give me some confidence though. Given that, I'm fine with this change going in. Hopefully #6274 will illuminate more as well as #6166.

Signed-off-by: ktf <krunotf@gmail.com>
@ktff
Copy link
Contributor Author

ktff commented Feb 4, 2021

@jszwedko

The fact that the improvement is consistent across all of them does give me some confidence though.

Indeed, in criterion it's more or less consistent, even all of 15 topology/tcp_socket benches are showing improvement, but the test harness is somewhat inconclusive, nightly seams better overall, but dev shows a persistent 2-3% improvement in net recv. Let's wait for #6274 to see what it will show.

Looking at this results, my main worry is, that since futures channels relies more on allocations, that it's faster in criterion benches since it doesn't have to clean up after itself/has more free clean memory/etc during the bench, but that's not true during test harness so it's slower or on par there.

@jszwedko
Copy link
Member

jszwedko commented Feb 4, 2021

I merged in master so what the new topology bench looks like.

@ktff
Copy link
Contributor Author

ktff commented Feb 5, 2021

Now we are talking

2021-02-04T22:08:55.7594172Z Benchmarking real_world_1/topology: Analyzing
2021-02-04T22:08:57.1529661Z real_world_1/topology   time:   [3.4942 s 3.4978 s 3.5015 s]
2021-02-04T22:08:57.1530289Z                         thrpt:  [28.559 Kelem/s 28.590 Kelem/s 28.619 Kelem/s]
2021-02-04T22:08:57.1530983Z                  change:
2021-02-04T22:08:57.1532055Z                         time:   [-9.8741% -9.7432% -9.6084%] (p = 0.00 < 0.05)
2021-02-04T22:08:57.1532589Z                         thrpt:  [+10.630% +10.795% +10.956%]

The new benchmark shows consistent 10% improvement, so I'll run this one more time for good measure since it's a new bench and then we can go ahead with this.

cc @jszwedko

Signed-off-by: ktf <krunotf@gmail.com>
@jszwedko
Copy link
Member

jszwedko commented Feb 5, 2021

Nice! 🎉

Signed-off-by: ktf <krunotf@gmail.com>
@ktff ktff marked this pull request as ready for review February 7, 2021 18:25
@ktff ktff requested review from a team and pablosichert and removed request for a team and pablosichert February 7, 2021 18:25
Signed-off-by: ktf <krunotf@gmail.com>
@ktff ktff merged commit 85c0a29 into master Feb 7, 2021
@ktff ktff deleted the ktff/future03_channel branch February 7, 2021 23:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: task Generic non-code related tasks
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Investigate performance regression from tokio channel upgrade
3 participants