Handle overflow in rd_buf_write_remains

[anchitj]

After the implementation of KIP-320, TopicCnt was sent as a varint in the metadata request. This change caused an overflow issue due to the use of rd_buf_erase to erase the remaining bytes. The overflow occurred in the calculation rbuf->rbuf_size - (rbuf->rbuf_len + rbuf->rbuf_erased) because rbuf->rbuf_erased was non-zero when the buffer was almost full, leading to an overflow condition.

As a result, for certain configurations that caused the buffer to be nearly full, the client was unable to send metadata requests, and it also caused crashes in the .NET client.

[korchak-aleksandr]

Hi @emasab,

Could you please review and approve this PR? It's critical for us as it addresses a blocking issue with the confluent-kafka-dotnet library update. Your approval is eagerly awaited.

[emasab]

Thanks Anchit, the source of the problem comes from a bug that happens previously, check the comment.

[emasab]

There also a problem in rd_buf_write_seek because of rd_buf_destroy_segment.

My proposed solution is to add a seg_erased to rd_segment_s and increase it when part of the segment is erased, then subtract seg_erased from rbuf_erased when a segment is removed in rd_buf_write_seek

[anchitj]

There also a problem in rd_buf_write_seek because of rd_buf_destroy_segment.

My proposed solution is to add a seg_erased to rd_segment_s and increase it when part of the segment is erased, then subtract seg_erased from rbuf_erased when a segment is removed in rd_buf_write_seek

Makes sense. I'll create a separate PR to handle that.

[emasab]

@anchitj better to do it here, as it's basically the same thing that is missing

[emasab]

Thanks Anchit, some changes are needed

[emasab]

Thanks Anchit! Let's wait for the CI and then we can merge this.

[nkostoulas]

Could this affect the confluent-kafka-go AdminClient?