Transaction performance

[jonathangenlambda]

I plan to use Kafka Transactions via the Haskell library hw-kafka-client, that builds on librdkafka, using the latest version 2.1.1. I conducted performance measurements for 2 different scenarios, running on a simple local Kafka environment, via docker compose, with 1 broker and 1 parition per topic. The way I measure is to count the throughput for each second (reset after 1000 milliseconds):

1. scenario is where a single transactional producer writes as fast as it can a single very small message (an integer) transactionally to Kafka topic Z. So the transaction involves writing a single message to topic Z.

• In case of a synchronous producer (flushing), I arrive at ~130 Transactions Per Second (TPS).
• With asynchronous producer (no flushing), it's over 800 TPS.

2. scenario has a non-transactional producer writing as fast as it can into a topic X. In a separate thread this topic X is transactionally consumed together with writing a single very small message (also an integer) transactionall to another Kafka topic Y. So the transaction involves reading a single message from topic X and writing a single message to topic Y.

• In both cases I arrive at quite unstable measurements, varying between 10 and 30 TPS.

I know that processing single messages in a transaction is not recommended, but still I am especially surprised by the low performance of the 2nd scenario, as didn't expect adding a transactional consume to drop the performance that severely.

I am wondering, why the performance of transactions is so low and what I could do to increase it? It might be a quirk of the Haskell hw-kafka-client library, but this is hard to tell - are there any reliable performance measurements on using transactions for single message processing using librdkafka?