We need a size limit for cache of single managedLeger

[KernLC]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

When a topic produce too much messages, the caches of broker will be full with this topic. In this case, the fetch request of other consumers cannot hit the cache and they only can read entries by bookeeper.

Solution

Add a size limit for cache of single managedLedger.

Alternatives

~

Anything else?

~

Are you willing to submit a PR?

• I'm willing to submit a PR!

[lhotari]

Yes, the current broker cache isn't optimal. Please participate in this dev mailing list discussion: https://lists.apache.org/thread/xm095hnjo0cffbdy8ckysmzzm90gsbnp .
There are mailing list joining instructions at https://pulsar.apache.org/contact/

Add a size limit for cache of single managedLedger.

Instead of a size limit, the caching algorithm should have a proper way to do optimal decisions and set the limits dynamically based on the load.

[lhotari]

There's also "PIP-174: Provide new implementation for broker dispatch cache" #15954 / #15955 to improve broker cache.