Low performance observed in Python writer

[RaulGracia]

Problem description

I have been testing the Python binding for Pravega. One of the things that caught my eye was the actual performance I could obtain of applications writing events when using standalone (in-memory) Pravega. After that, I decided to do an actual micro-benchmark for the Python writer consisting of writing as fast as possible the string "hello world" to a 1-segment stream:

Written 100000 events in 38.94435238838196 seconds
Written 100000 events in 38.72876286506653 seconds
Written 100000 events in 39.33560848236084 seconds

As can be observed, the Python writer consistently gets around 2500 to 2600 events/second.

If we compare this result with a similar benchmark in Java, we can observe the following (using Pravega Benchmark):

bin/pravega-benchmark -producers 1  -scope test -segments 1 -stream benchmark -size 20 -events 500000 -time 20 -controller "tcp://localhost:9090"
2023-07-04 17:00:32:970 +0200 [ForkJoinPool-1-worker-3] INFO io.pravega.perf.PerfStats - 9334882 records Writing, 472174.102 records/sec, 20 bytes record size, 9.01 MiB/sec, 681.3 ms avg latency, 3484.0 ms max latency, 11 ms 50th, 1203 ms 75th, 3139 ms 95th, 3316 ms 99th, 3398 ms 99.9th, 3410 ms 99.99th.

While the Pravega standalone gets out of memory quick when inducing high throughput, the point of this issue is that we can get around half a million (or more if we increased the memory settings) events per second. This is a throughput difference of 200x comparing the Python and Java clients for Pravega.

Problem location
Python binding for Pravega.

Suggestions for an improvement
We need to understand if the problem lies in the Rust implementation of the client or if it is some intrinsic performance hit of language bindings.

[thekingofcity]

Not sure how you benchmark the Python client. Default settings will create a writer that writes the next event only after it receives the acknowledgment from the server.

pravega-client-rust/python/src/stream_manager.rs

Line 407 in 73b46a6

/// By default the max inflight events is configured for 0. The users can change this value

When compared with the same settings, the Python client should be slower no more than 10x.

[RaulGracia]

Thanks a lot @thekingofcity, I was not familiar with the max_inflight_events events option in the client. And you are right, it totally changes the picture in terms of performance, at least with the micro-benchmark I used. These are the results changing the value of max_inflight_events:

• max_inflight_events=10: 18,900 events/second

Written 1000000 events in 53.04908895492554 seconds 

• max_inflight_events=100: 150,000 events/second

Written 1000000 events in 6.702754497528076 seconds

• max_inflight_events=1000: 545,000 events/second

Written 1000000 events in 1.8340561389923096 seconds

• max_inflight_events=10000: 637,000 events/second

Written 1000000 events in 1.5698730945587158 seconds

I wonder if it would be reasonable to set the max_inflight_events default value to something larger than 0, as otherwise people testing the Python binding may fall into the same error as me. Is there any specific reason to not using something like max_inflight_events=1000 as a default? Is there any implication on latency on changing this value?

[RaulGracia]

I will close this issue, as the problem was related to the configuration. @thekingofcity @tkaitchuck just consider if it would be reasonable setting another default value for max_inflight_events that could provide higher throughput by default. Thanks!

[thekingofcity]

@tkaitchuck does have an issue improving write on the Python client. See #352 :)