Add thrashing of old txn sessions to prevent OOM

[rystsov]

PR adds tx thrashing to abort & evict old txn sessions once the number of known txn session passes a threshold (new max_transactions_per_coordinator property).

Fixes #12424

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v23.2.x
• v23.1.x
• v22.3.x

Release Notes

• none

[rystsov]

/ci-repeat 5
dt-repeat=1

[rystsov]

/ci-repeat 5
dt-repeat=1

[rystsov]

/ci-repeat 5
dt-repeat=1

[rystsov]

/ci-repeat 5
dt-repeat=1

[rystsov]

/ci-repeat 5
dt-repeat=1

[mmaslankaprv]

I am wondering if in the case when tx_id limit is breached we should simply return an error until old transactions are expired completed ? Also why other Kafka protocol implementations don't have a limit for max concurrent transactions ?

[rystsov]

I am wondering if in the case when tx_id limit is breached we should simply return an error until old transactions are expired completed?

We can't rely on expiration because it's controlled by the users and Flink recommends to set it to pretty large values

Also why other Kafka protocol implementations don't have a limit for max concurrent transactions?

They don't have the limit but they have the same problem: see KIP-936. Once it's approved and implemented we'll support to be compatible but we can't wait and to fix existing problem we'll use the limit so far.

[rystsov]

Known issue: #12659

[mmaslankaprv]

I am wondering if in the case when tx_id limit is breached we should simply return an error until old transactions are expired completed?

We can't rely on expiration because it's controlled by the users and Flink recommends to set it to pretty large values

Also why other Kafka protocol implementations don't have a limit for max concurrent transactions?

They don't have the limit but they have the same problem: see KIP-936. Once it's approved and implemented we'll support to be compatible but we can't wait and to fix existing problem we'll use the limit so far.

The KIP-936 is mentioning producer_id not transactional_id so it is still not clear to me how this will help ?

[rystsov]

I am wondering if in the case when tx_id limit is breached we should simply return an error until old transactions are expired completed?

We can't rely on expiration because it's controlled by the users and Flink recommends to set it to pretty large values

Also why other Kafka protocol implementations don't have a limit for max concurrent transactions?

They don't have the limit but they have the same problem: see KIP-936. Once it's approved and implemented we'll support to be compatible but we can't wait and to fix existing problem we'll use the limit so far.

The KIP-936 is mentioning producer_id not transactional_id so it is still not clear to me how this will help?

Reread it, yeah it mentioned only producer id caused OOMs. Maybe they'll introduce quotas to txn coordinator too

[rystsov]

Only known issues fail:

• ShutdownTest.test_timely_shutdown_with_failures CI Failure (NodeCrash) in ShutdownTest.test_timely_shutdown_with_failures #12659
• cloud_storage_rpfixture CI Failure (unknown error) in cloud_storage_rpfixture #12120

[vbotbuildovich]

/backport v23.2.x

[vbotbuildovich]

/backport v23.1.x

[vbotbuildovich]

Failed to run cherry-pick command. I executed the commands below:

git checkout -b backport-pr-12477-v23.1.x-248 remotes/upstream/v23.1.x
git cherry-pick -x cdd6481925721ab10f26aedb52d0318038762eca 6e4fce3b81a23f58cf0105fa0afd0ceacfc9a0ea 679baa6179bbac06312636848aca33fd0c87519f 48d85d29c57280d5167889fae3bcaac5e1867336 03d4a0e2798d87bc565b341b944930fcb3fb82e1 b4ed101182bafa92609c6be86e8515c59455fce7 37d745f3a85d85edc324e7739439af62388edbdb dc9b56ec32b1a648b3ea262987de4c5d54f37572 4bf7d351654cbe4bce41a7562bba3b466e44f5ce 1da806427e71f3888dcf03257a3552d1a94978cc b831fd8ad5bf00b43e7a39228ba0f3a02a0c10e2

Workflow run logs.