You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello, we experience an odd behaviour when testing the batch behaviour of the library. We are producing 10K records in parallel with ProduceSync and with MaxBufferedRecords 5000. We would expect that we will produce 2 batches. Instead, we see a huge delay in producing. With bigger numbers it takes forever. This can be produced in all cases where MaxBufferedRecords is smaller than the count of produced records in parallel. I was not able to reproduce the same behaviour with Produce.
Here is an overview to get an impression of the block behaviour
because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 8.125µs, time_to_read: 595.292µs, err: <nil>
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 518.917µs, time_to_read: 2.792µs, err: <nil>
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 439.584µs, time_to_read: 5.416µs, err: <nil>
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15724=>15740}]
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15740=>15741}]
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15741=>15742}]
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 400.417µs, time_to_read: 30.666µs, err: <nil>
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15742=>15743}]
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 486, write_wait: 43.875µs, time_to_write: 5.458µs, err: <nil>
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 455, write_wait: 41.375µs, time_to_write: 9.292µs, err: <nil>
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 455, write_wait: 45.167µs, time_to_write: 8.292µs, err: <nil>
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 455, write_wait: 31.167µs, time_to_write: 4µs, err: <nil>
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 53.084µs, time_to_read: 851.208µs, err: <nil>
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 868.875µs, time_to_read: 1.75µs, err: <nil>
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 758.25µs, time_to_read: 1.25µs, err: <nil>
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 457.25µs, time_to_read: 5.375µs, err: <nil>
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15743=>15750}]
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15750=>15751}]
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15751=>15752}]
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15752=>15753}]
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 455, write_wait: 62.834µs, time_to_write: 7.416µs, err: <nil>
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 455, write_wait: 45.25µs, time_to_write: 12.25µs, err: <nil>
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 455, write_wait: 52.167µs, time_to_write: 6.5µs, err: <nil>
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 455, write_wait: 58.958µs, time_to_write: 5.375µs, err: <nil>
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 6.792µs, time_to_read: 1.823625ms, err: <nil>
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 1.759333ms, time_to_read: 6.042µs, err: <nil>
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 1.568375ms, time_to_read: 1.208µs, err: <nil>
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 1.475625ms, time_to_read: 6.542µs, err: <nil>
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15753=>15754}]
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15754=>15755}]
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15755=>15756}]
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15756=>15757}]
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 531, write_wait: 129.333µs, time_to_write: 6.167µs, err: <nil>
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 455, write_wait: 86.583µs, time_to_write: 13.542µs, err: <nil>
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 59.125µs, time_to_read: 297.083µs, err: <nil>
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15757=>15774}]
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 312.417µs, time_to_read: 101.041µs, err: <nil>
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 455, write_wait: 43.208µs, time_to_write: 5µs, err: <nil>
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15774=>15775}]
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 455, write_wait: 36.208µs, time_to_write: 13.584µs, err: <nil>
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 455, write_wait: 206.792µs, time_to_write: 12.25µs, err: <nil>
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 455, write_wait: 182.666µs, time_to_write: 17.459µs, err: <nil>
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 51.333µs, time_to_read: 1.444209ms, err: <nil>
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 1.305166ms, time_to_read: 1.584µs, err: <nil>
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 1.076666ms, time_to_read: 1.334µs, err: <nil>
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 1.0355ms, time_to_read: 6.5µs, err: <nil>
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15775=>15776}]
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15776=>15777}]
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15777=>15778}]
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15778=>15779}]
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 482, write_wait: 57.042µs, time_to_write: 8.041µs, err: <nil>
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 455, write_wait: 45.167µs, time_to_write: 7.583µs, err: <nil>
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 455, write_wait: 33.583µs, time_to_write: 6.5µs, err: <nil>
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 455, write_wait: 38.417µs, time_to_write: 7.458µs, err: <nil>
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 9.417µs, time_to_read: 1.586042ms, err: <nil>
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 1.2085ms, time_to_read: 1.875µs, err: <nil>
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 1.025ms, time_to_read: 1.542µs, err: <nil>
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 812.833µs, time_to_read: 7.375µs, err: <nil>
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15779=>15785}]
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15785=>15786}]
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15786=>15787}]
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15787=>15788}]
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 464, write_wait: 56.791µs, time_to_write: 5.542µs, err: <nil>
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 455, write_wait: 72µs, time_to_write: 7.375µs, err: <nil>
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 42.708µs, time_to_read: 253.875µs, err: <nil>
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15788=>15790}]
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 272.75µs, time_to_read: 85.083µs, err: <nil>
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15790=>15791}]
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 455, write_wait: 98.25µs, time_to_write: 17.916µs, err: <nil>
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 455, write_wait: 42.958µs, time_to_write: 7.375µs, err: <nil>
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 24.209µs, time_to_read: 464.625µs, err: <nil>
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15791=>15792}]
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 207µs, time_to_read: 153.25µs, err: <nil>
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15792=>15793}]
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 455, write_wait: 383.417µs, time_to_write: 8.75µs, err: <nil>
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 455, write_wait: 430.375µs, time_to_write: 10.75µs, err: <nil>
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 455, write_wait: 519.625µs, time_to_write: 10.792µs, err: <nil>
[DEBUG] read Produce v11; broker: 1, bytes_read: 70, read_wait: 405.541µs, time_to_read: 2.334µs, err: <nil>
[DEBUG] produced; broker: 1, to: 3055433056991211251[0{15793=>15794}]
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 556, write_wait: 74.334µs, time_to_write: 7.583µs, err: <nil>
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] wrote Produce v11; broker: 1, bytes_written: 455, write_wait: 42.834µs, time_to_write: 5.833µs, err: <nil>
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
[DEBUG] Produce block awoken, we now have space to produce, continuing to partition and produce
[DEBUG] blocking Produce because we are either over max buffered records or max buffered bytes; over_max_records: true, over_max_bytes: false
.....
PASS
Debugger finished with the exit code 0
The text was updated successfully, but these errors were encountered:
Hello, we experience an odd behaviour when testing the batch behaviour of the library. We are producing 10K records in parallel with
ProduceSync
and withMaxBufferedRecords
5000. We would expect that we will produce 2 batches. Instead, we see a huge delay in producing. With bigger numbers it takes forever. This can be produced in all cases whereMaxBufferedRecords
is smaller than the count of produced records in parallel. I was not able to reproduce the same behaviour withProduce
.Here is an overview to get an impression of the block behaviour
ProduceSync 10K, MaxBufferedRecords: 10K = 700ms
ProduceSync 10K, MaxBufferedRecords: 9K = 3s
ProduceSync 10K, MaxBufferedRecords: 9K, with -race flag = 40s
Pseudocode. Each topic has 1 partition and replication factor of 2.
Hardware: Apple M3 Max
The Logs. The repetitive pattern never ends.
The text was updated successfully, but these errors were encountered: