Skip to content

Akka.Benchmarks: run real duplex messaging over Akka.IO.Tcp #7651

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 7 commits into from
May 15, 2025
Merged

Akka.Benchmarks: run real duplex messaging over Akka.IO.Tcp #7651

merged 7 commits into from
May 15, 2025

Conversation

@Aaronontheweb
Copy link
Member

@Aaronontheweb Aaronontheweb commented May 15, 2025 •
edited
Loading

Changes

This is a more realistic test of how fast the system can go, is more similar to real world usage, and is also what we do inside RemotePingPong

Checklist

For significant changes, please ensure that the following have been completed (delete if not relevant):

Latest dev Benchmarks

BenchmarkDotNet v0.13.12, Pop!_OS 22.04 LTS
13th Gen Intel Core i7-1360P, 1 CPU, 16 logical and 12 physical cores
.NET SDK 8.0.404
  [Host]  : .NET 8.0.11 (8.0.1124.51707), X64 RyuJIT AVX2
  LongRun : .NET 8.0.11 (8.0.1124.51707), X64 RyuJIT AVX2

Job=LongRun  Concurrent=True  Server=True
InvocationCount=1  IterationCount=10  LaunchCount=3
RunStrategy=Monitoring  UnrollFactor=1  WarmupCount=3
Method MessageLength ClientsCount Mean Error StdDev Req/sec
ClientServerCommunication 10 1 28.126 us 0.9439 us 1.4129 us 35,554.20
ClientServerCommunication 10 3 14.388 us 2.1218 us 3.1758 us 69,502.83
ClientServerCommunication 10 5 11.058 us 0.9752 us 1.4597 us 90,435.79
ClientServerCommunication 10 7 8.393 us 0.5154 us 0.7715 us 119,148.80
ClientServerCommunication 10 10 6.226 us 0.4926 us 0.7372 us 160,626.16
ClientServerCommunication 10 20 4.470 us 0.4500 us 0.6735 us 223,712.48
ClientServerCommunication 10 30 4.412 us 0.4499 us 0.6734 us 226,655.74
ClientServerCommunication 10 40 4.270 us 0.4029 us 0.6031 us 234,208.44
ClientServerCommunication 100 1 24.766 us 0.8864 us 1.3267 us 40,378.29
ClientServerCommunication 100 3 13.921 us 2.0134 us 3.0135 us 71,833.61
ClientServerCommunication 100 5 11.419 us 0.8069 us 1.2077 us 87,572.40
ClientServerCommunication 100 7 8.471 us 0.4339 us 0.6494 us 118,049.74
ClientServerCommunication 100 10 6.204 us 0.4274 us 0.6397 us 161,194.43
ClientServerCommunication 100 20 4.123 us 0.3477 us 0.5204 us 242,515.72
ClientServerCommunication 100 30 4.584 us 0.7039 us 1.0536 us 218,138.99
ClientServerCommunication 100 40 4.176 us 0.3487 us 0.5219 us 239,474.87

This PR's Benchmarks

BenchmarkDotNet v0.13.12, Pop!_OS 22.04 LTS
13th Gen Intel Core i7-1360P, 1 CPU, 16 logical and 12 physical cores
.NET SDK 8.0.404
  [Host]  : .NET 8.0.11 (8.0.1124.51707), X64 RyuJIT AVX2
  LongRun : .NET 8.0.11 (8.0.1124.51707), X64 RyuJIT AVX2

Job=LongRun  Concurrent=True  Server=True
InvocationCount=1  IterationCount=10  LaunchCount=3
RunStrategy=Monitoring  UnrollFactor=1  WarmupCount=3
Method MessageLength ClientsCount Mean Error StdDev Req/sec
ClientServerCommunication 10 1 27.580 us 1.4212 us 2.1273 us 36,258.00
ClientServerCommunication 10 3 14.604 us 2.8916 us 4.3280 us 68,474.99
ClientServerCommunication 10 5 10.568 us 1.0643 us 1.5930 us 94,629.43
ClientServerCommunication 10 7 8.305 us 0.5406 us 0.8092 us 120,406.16
ClientServerCommunication 10 10 6.416 us 0.5650 us 0.8457 us 155,870.32
ClientServerCommunication 10 20 4.676 us 0.6057 us 0.9067 us 213,846.83
ClientServerCommunication 10 30 4.537 us 0.6251 us 0.9356 us 220,414.60
ClientServerCommunication 10 40 4.323 us 0.3606 us 0.5398 us 231,317.10
ClientServerCommunication 100 1 24.559 us 1.0431 us 1.5612 us 40,719.05
ClientServerCommunication 100 3 14.427 us 2.2594 us 3.3817 us 69,315.43
ClientServerCommunication 100 5 10.264 us 1.0891 us 1.6301 us 97,428.32
ClientServerCommunication 100 7 8.062 us 0.5350 us 0.8007 us 124,034.27
ClientServerCommunication 100 10 6.339 us 0.4514 us 0.6757 us 157,748.27
ClientServerCommunication 100 20 4.491 us 0.4884 us 0.7311 us 222,675.95
ClientServerCommunication 100 30 4.170 us 0.5644 us 0.8448 us 239,827.56
ClientServerCommunication 100 40 4.235 us 0.4853 us 0.7264 us 236,104.87

@Aaronontheweb
Akka.Benchmarks: run real duplex messaging over Akka.IO.Tcp
795f9e7 
This is a more realistic test of how fast the system can go
@Aaronontheweb
Copy link
Member Author

Aaronontheweb commented May 15, 2025

The changes had very little impact to the observed performance - which either tells us that we have a massive flow control bottle neck inside the Tcp actors themselves or we have significant headroom above what the underlying TCP system can deliver. Won't be able to tell until we run this on #7637

@Aaronontheweb Aaronontheweb marked this pull request as ready for review May 15, 2025 18:32
Aaronontheweb
Aaronontheweb commented May 15, 2025
View reviewed changes
src/benchmark/Akka.Benchmarks/IO/TcpOperationsBenchmarks.cs
Sender.Tell(Tcp.Write.Create(ByteString.FromBytes(message)));

// send messages in bursts of 20
for (var i = 0; i < 20; i++)
Copy link
Member Author

@Aaronontheweb Aaronontheweb May 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prime 20 messages in-flight at the same time

@Aaronontheweb
Copy link
Member Author

Aaronontheweb commented May 15, 2025

Actually, the problem is that we're not doing any real message framing - therefore it's totally possible that all 20 messages just get absorbed into a single big message by the second or third time the EchoActor gets its Received

@Aaronontheweb
Copy link
Member Author

Aaronontheweb commented May 15, 2025

Got it working with message framing now and the results are looking a lot more like what I'd expect to see

@Aaronontheweb
Copy link
Member Author

Aaronontheweb commented May 15, 2025

Results

BenchmarkDotNet v0.13.12, Pop!_OS 22.04 LTS                                                                                                        
13th Gen Intel Core i7-1360P, 1 CPU, 16 logical and 12 physical cores                                                                              
.NET SDK 8.0.404                                                                                                                                   
  [Host]  : .NET 8.0.11 (8.0.1124.51707), X64 RyuJIT AVX2                                                                                          
  LongRun : .NET 8.0.11 (8.0.1124.51707), X64 RyuJIT AVX2                                                                                          
                                                                                                                                                   
Job=LongRun  Concurrent=True  Server=True                                                                                                          
InvocationCount=1  IterationCount=10  LaunchCount=3                                                                                                
RunStrategy=Monitoring  UnrollFactor=1  WarmupCount=3
Method MessageLength ClientsCount Mean Error StdDev Median Req/sec Gen0 Allocated
ClientServerCommunication 10 1 NA NA NA NA NA NA
ClientServerCommunication 10 3 1,456.3 ns 107.00 ns 160.15 ns 1,428.8 ns 686,684.20 0.0060 1220 B
ClientServerCommunication 10 5 1,023.4 ns 129.51 ns 193.84 ns 898.2 ns 977,160.80 0.0060 1178 B
ClientServerCommunication 10 7 799.7 ns 123.16 ns 184.34 ns 684.5 ns 1,250,449.79 0.0060 1166 B
ClientServerCommunication 10 10 691.7 ns 80.62 ns 120.66 ns 708.0 ns 1,445,760.35 0.0060 1092 B
ClientServerCommunication 10 20 414.7 ns 63.02 ns 94.33 ns 364.5 ns 2,411,488.57 0.0050 1001 B
ClientServerCommunication 10 30 398.6 ns 49.98 ns 74.81 ns 365.3 ns 2,508,648.33 0.0050 965 B
ClientServerCommunication 10 40 470.4 ns 79.47 ns 118.95 ns 472.9 ns 2,125,639.64 0.0050 953 B
ClientServerCommunication 100 1 NA NA NA NA NA NA
ClientServerCommunication 100 3 1,624.8 ns 124.27 ns 186.00 ns 1,604.8 ns 615,447.22 0.0080 1437 B
ClientServerCommunication 100 5 1,153.5 ns 143.15 ns 214.26 ns 1,016.4 ns 866,951.62 0.0070 1430 B
ClientServerCommunication 100 7 831.3 ns 97.25 ns 145.56 ns 762.0 ns 1,202,920.19 0.0070 1425 B
ClientServerCommunication 100 10 745.2 ns 156.58 ns 234.36 ns 647.8 ns 1,341,927.12 0.0070 1384 B
ClientServerCommunication 100 20 516.2 ns 57.83 ns 86.56 ns 534.6 ns 1,937,118.40 0.0070 1283 B
ClientServerCommunication 100 30 526.4 ns 66.69 ns 99.82 ns 564.3 ns 1,899,851.58 0.0070 1262 B
ClientServerCommunication 100 40 416.9 ns 46.28 ns 69.27 ns 387.6 ns 2,398,438.10 0.0070 1251 B

@Aaronontheweb
Copy link
Member Author

Aaronontheweb commented May 15, 2025

That's more like it, but there's definitely a bug when we're running with only 1 client

@Aaronontheweb
Copy link
Member Author

Aaronontheweb commented May 15, 2025

Here we go - final numbers:

Final Numbers

BenchmarkDotNet v0.13.12, Pop!_OS 22.04 LTS                                                                                                
13th Gen Intel Core i7-1360P, 1 CPU, 16 logical and 12 physical cores                                                                      
.NET SDK 8.0.404                                                                                                                           
  [Host]  : .NET 8.0.11 (8.0.1124.51707), X64 RyuJIT AVX2                                                                                  
  LongRun : .NET 8.0.11 (8.0.1124.51707), X64 RyuJIT AVX2                                                                                  
                                                                                                                                           
Job=LongRun  Concurrent=True  Server=True                                                                                                  
InvocationCount=1  IterationCount=10  LaunchCount=3                                                                                        
RunStrategy=Monitoring  UnrollFactor=1  WarmupCount=3
Method MessageLength ClientsCount Mean Error StdDev Req/sec Gen0 Allocated
ClientServerCommunication 10 1 5,135.3 ns 236.85 ns 354.51 ns 194,731.42 0.0060 1206 B
ClientServerCommunication 10 3 1,478.6 ns 111.37 ns 166.69 ns 676,327.57 0.0060 1221 B
ClientServerCommunication 10 5 1,124.8 ns 128.99 ns 193.06 ns 889,068.01 0.0060 1170 B
ClientServerCommunication 10 7 825.4 ns 104.25 ns 156.03 ns 1,211,471.51 0.0060 1150 B
ClientServerCommunication 10 10 639.7 ns 113.02 ns 169.16 ns 1,563,153.36 0.0060 1102 B
ClientServerCommunication 10 20 371.2 ns 31.36 ns 46.94 ns 2,694,289.82 0.0050 998 B
ClientServerCommunication 10 30 414.9 ns 65.05 ns 97.37 ns 2,410,263.95 0.0050 963 B
ClientServerCommunication 10 40 421.7 ns 68.71 ns 102.84 ns 2,371,095.25 0.0050 955 B
ClientServerCommunication 100 1 5,526.0 ns 359.47 ns 538.04 ns 180,961.13 0.0070 1437 B
ClientServerCommunication 100 3 1,672.0 ns 156.09 ns 233.63 ns 598,092.81 0.0080 1433 B
ClientServerCommunication 100 5 1,204.7 ns 150.04 ns 224.57 ns 830,072.77 0.0070 1431 B
ClientServerCommunication 100 7 869.2 ns 111.72 ns 167.21 ns 1,150,541.82 0.0070 1431 B
ClientServerCommunication 100 10 675.4 ns 104.88 ns 156.97 ns 1,480,676.62 0.0070 1385 B
ClientServerCommunication 100 20 601.2 ns 60.82 ns 91.03 ns 1,663,267.73 0.0070 1276 B
ClientServerCommunication 100 30 497.8 ns 75.23 ns 112.60 ns 2,008,798.91 0.0070 1258 B
ClientServerCommunication 100 40 449.2 ns 63.37 ns 94.86 ns 2,226,409.78 0.0060 1253 B

@Aaronontheweb Aaronontheweb merged commit b685b27 into akkadotnet:dev May 15, 2025
6 of 11 checks passed
@Aaronontheweb Aaronontheweb deleted the Akka.IO.Tcp-benchmarks-non-block branch May 15, 2025 20:56
@Aaronontheweb Aaronontheweb mentioned this pull request May 15, 2025
Merged
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

akka-io perf

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Aaronontheweb