Skip to content

Rust benchmarks #21

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
uatuko opened this issue Jan 14, 2024 · 2 comments
Closed

Rust benchmarks #21

uatuko opened this issue Jan 14, 2024 · 2 comments
Assignees
@uatuko
Labels
documentation Improvements or additions to documentation

Comments

@uatuko
Copy link
Owner

uatuko commented Jan 14, 2024

Rust is gaining popularity as a reliable and an efficient language so it would be interesting to see how a rust gRPC server compares in benchmarks.
@uatuko uatuko added the documentation Improvements or additions to documentation label Jan 14, 2024
@uatuko
Copy link
Owner Author

uatuko commented Jan 14, 2024

Rust is not an officially supported language according to https://grpc.io/docs/languages/ but after a quick search it seems tonic is a popular implementation.

@uatuko uatuko self-assigned this Jan 14, 2024
@uatuko
Copy link
Owner Author

uatuko commented Jan 14, 2024

Following benchmarks were run using tonic v0.10.2 helloworld example.

💡This log line was commented out to get comparable results.

❯ rustc --version
rustc 1.71.1 (eb26296b5 2023-08-03) (built from a source tarball)

❯ cargo --version
cargo 1.71.2

❯ cargo run --release --bin helloworld-server
    Finished release [optimized] target(s) in 0.20s
     Running `/path/to/hyperium/tonic/target/release/helloworld-server`
GreeterServer listening on [::1]:50051

Smoke tests

❯ grpcurl -plaintext -import-path ./proto/helloworld -proto helloworld.proto -d '{"name": "Tonic"}' '[::1]:50051' helloworld.Greeter/SayHello
{
  "message": "Hello Tonic!"
}

❯ nghttp --verbose --header='content-type: application/grpc' --header='te: trailers' --data=tmp/helloworld.Greeter_SayHello.request.data http://localhost:50051/helloworld.Greeter/SayHello
[  0.002] Connected
[  0.002] send SETTINGS frame <length=12, flags=0x00, stream_id=0>
          (niv=2)
          [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
          [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65535]
[  0.002] send PRIORITY frame <length=5, flags=0x00, stream_id=3>
          (dep_stream_id=0, weight=201, exclusive=0)
[  0.002] send PRIORITY frame <length=5, flags=0x00, stream_id=5>
          (dep_stream_id=0, weight=101, exclusive=0)
[  0.002] send PRIORITY frame <length=5, flags=0x00, stream_id=7>
          (dep_stream_id=0, weight=1, exclusive=0)
[  0.002] send PRIORITY frame <length=5, flags=0x00, stream_id=9>
          (dep_stream_id=7, weight=1, exclusive=0)
[  0.002] send PRIORITY frame <length=5, flags=0x00, stream_id=11>
          (dep_stream_id=3, weight=1, exclusive=0)
[  0.002] send HEADERS frame <length=90, flags=0x24, stream_id=13>
          ; END_HEADERS | PRIORITY
          (padlen=0, dep_stream_id=11, weight=16, exclusive=0)
          ; Open new stream
          :method: POST
          :path: /helloworld.Greeter/SayHello
          :scheme: http
          :authority: localhost:50051
          accept: */*
          accept-encoding: gzip, deflate
          user-agent: nghttp2/1.58.0
          content-length: 18
          content-type: application/grpc
          te: trailers
[  0.002] send DATA frame <length=18, flags=0x01, stream_id=13>
          ; END_STREAM
[  0.002] recv SETTINGS frame <length=18, flags=0x00, stream_id=0>
          (niv=3)
          [SETTINGS_INITIAL_WINDOW_SIZE(0x04):1048576]
          [SETTINGS_MAX_FRAME_SIZE(0x05):16384]
          [SETTINGS_MAX_HEADER_LIST_SIZE(0x06):16777216]
[  0.002] send SETTINGS frame <length=0, flags=0x01, stream_id=0>
          ; ACK
          (niv=0)
[  0.002] recv SETTINGS frame <length=0, flags=0x01, stream_id=0>
          ; ACK
          (niv=0)
[  0.002] recv WINDOW_UPDATE frame <length=4, flags=0x00, stream_id=0>
          (window_size_increment=983041)
[  0.002] recv (stream_id=13) :status: 200
[  0.002] recv (stream_id=13) content-type: application/grpc
[  0.002] recv (stream_id=13) date: Sun, 14 Jan 2024 09:30:14 GMT
[  0.002] recv HEADERS frame <length=38, flags=0x04, stream_id=13>
          ; END_HEADERS
          (padlen=0)
          ; First response header

Hello grpc-client![  0.002] recv DATA frame <length=25, flags=0x00, stream_id=13>
[  0.002] recv (stream_id=13) grpc-status: 0
[  0.002] recv HEADERS frame <length=12, flags=0x05, stream_id=13>
          ; END_STREAM | END_HEADERS
          (padlen=0)
[  0.002] send GOAWAY frame <length=8, flags=0x00, stream_id=0>
          (last_stream_id=0, error_code=NO_ERROR(0x00), opaque_data(0)=[])

Benchmarks

Benchmarks were run on a MacBook Pro 2021 (M1 Max, 32GB), macOS 14.2.1 (23C71) and directly comparable to the results in #20.

💡Please refer to #2 for more info on benchmark scenarios.

1a 1b 2a 2b 3a 3b
Rust (tonic v0.10.2) 29k 66k 95k 176k 68k 212k

1a

❯ h2load --clients=1 --requests=100000 --header='Content-Type: application/grpc' --header='te: trailers' --data=tmp/helloworld.Greeter_SayHello.request.data http://localhost:50051/helloworld.Greeter/SayHello 
starting benchmark...
spawning thread #0: 1 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 3.50s, 28606.80 req/s, 1.53MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.34MB (5600229) total, 293.07KB (300104) headers (space savings 95.77%), 2.38MB (2500000) data
                     min         max         mean         sd        +/- sd
time for request:       23us       258us        32us         6us    91.75%
time for connect:      645us       645us       645us         0us   100.00%
time to 1st byte:      916us       916us       916us         0us   100.00%
req/s           :   28606.83    28606.83    28606.83        0.00   100.00%

1b

❯ h2load --clients=1 --requests=100000 --max-concurrent-streams=10 --header='Content-Type: application/grpc' --header='te: trailers' --data=tmp/helloworld.Greeter_SayHello.request.data http://localhost:50051/helloworld.Greeter/SayHello
starting benchmark...
spawning thread #0: 1 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 1.51s, 66323.77 req/s, 3.54MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.34MB (5600183) total, 293.03KB (300058) headers (space savings 95.77%), 2.38MB (2500000) data
                     min         max         mean         sd        +/- sd
time for request:       31us       789us       145us        53us    66.83%
time for connect:      535us       535us       535us         0us   100.00%
time to 1st byte:     1.09ms      1.09ms      1.09ms         0us   100.00%
req/s           :   66323.98    66323.98    66323.98        0.00   100.00%

2a

❯ h2load --clients=10 --requests=100000 --header='Content-Type: application/grpc' --header='te: trailers' --data=tmp/helloworld.Greeter_SayHello.request.data http://localhost:50051/helloworld.Greeter/SayHello
starting benchmark...
spawning thread #0: 10 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 1.05s, 95185.43 req/s, 5.08MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.34MB (5601200) total, 293.55KB (300600) headers (space savings 95.77%), 2.38MB (2500000) data
                     min         max         mean         sd        +/- sd
time for request:       24us       840us        88us        33us    76.65%
time for connect:     1.04ms      1.40ms      1.29ms       162us    70.00%
time to 1st byte:     1.83ms      2.22ms      2.06ms       144us    60.00%
req/s           :    9523.33     9554.54     9538.00        8.46    80.00%

2b

❯ h2load --clients=10 --requests=100000 --max-concurrent-streams=10 --header='Content-Type: application/grpc' --header='te: trailers' --data=tmp/helloworld.Greeter_SayHello.request.data http://localhost:50051/helloworld.Greeter/SayHello
starting benchmark...
spawning thread #0: 10 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 569.77ms, 175509.11 req/s, 9.37MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.34MB (5600950) total, 293.31KB (300350) headers (space savings 95.77%), 2.38MB (2500000) data
                     min         max         mean         sd        +/- sd
time for request:       70us      2.28ms       474us       229us    78.47%
time for connect:      772us      1.08ms       869us       105us    80.00%
time to 1st byte:     1.52ms      2.38ms      1.92ms       297us    70.00%
req/s           :   17570.16    17728.80    17626.04       53.41    70.00%

3a

❯ h2load --clients=100 --requests=100000 --header='Content-Type: application/grpc' --header='te: trailers' --data=tmp/helloworld.Greeter_SayHello.request.data http://localhost:50051/helloworld.Greeter/SayHello
starting benchmark...
spawning thread #0: 100 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 1.47s, 67938.43 req/s, 3.64MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.35MB (5611900) total, 298.73KB (305900) headers (space savings 95.69%), 2.38MB (2500000) data
                     min         max         mean         sd        +/- sd
time for request:       54us      7.85ms      1.05ms       368us    84.95%
time for connect:     4.61ms      9.42ms      7.57ms      1.54ms    61.00%
time to 1st byte:    12.56ms     14.10ms     13.52ms       428us    70.00%
req/s           :     680.09      682.65      681.28        0.61    63.00%

3b

❯ h2load --clients=100 --requests=100000 --max-concurrent-streams=10 --header='Content-Type: application/grpc' --header='te: trailers' --data=tmp/helloworld.Greeter_SayHello.request.data http://localhost:50051/helloworld.Greeter/SayHello
starting benchmark...
spawning thread #0: 100 total client(s). 100000 total requests
Application protocol: h2c
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 471.47ms, 212103.47 req/s, 11.35MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 5.35MB (5609500) total, 296.39KB (303500) headers (space savings 95.73%), 2.38MB (2500000) data
                     min         max         mean         sd        +/- sd
time for request:      112us     15.26ms      2.63ms      1.46ms    91.91%
time for connect:     4.03ms      8.45ms      7.34ms      1.19ms    86.00%
time to 1st byte:    11.61ms     23.23ms     18.02ms      3.88ms    53.00%
req/s           :    2124.30     2298.89     2160.77       31.85    81.00%

@uatuko uatuko closed this as completed Jan 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@uatuko uatuko

Labels
documentation Improvements or additions to documentation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@uatuko