Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve][fn] Make producer cache bounded and expiring in Functions/Connectors #22945

Merged
merged 4 commits into from
Jun 25, 2024

Conversation

lhotari
Copy link
Member

@lhotari lhotari commented Jun 19, 2024

Motivation

In Pulsar Functions and Pulsar IO Connectors, there's a Pulsar Producer cache, that
doesn't expire the producers and doesn't have a maximum size.
This is a problem for functions/connectors that produce to a large number of topics.

Modifications

  • Extract common ProducerCache
    • common Producer cache with separate cache areas for
      • producers created by calling Context, SinkContext, SourceContext methods
      • producers created in Pulsar Sources
        • multiple topics are possible by returning destination topics by SinkRecord.getDestinationTopic call
  • Keep backwards compatibility on the cache key
    • useThreadLocalProducers will use the Thread's ID as part of the key so that each thread has it's own isolated producer
    • In Sources, the SinkRecord.getPartitionId() is used as part of the cache key
  • Cache size is limited by the number of partitions in the producer
    • Default limit of the cache is 10000 partitions in total. Older entries will get removed from the cache if the cache overflows
  • Default cache entry expiration in 300 seconds after last access

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

@lhotari lhotari added this to the 3.4.0 milestone Jun 19, 2024
@lhotari lhotari self-assigned this Jun 19, 2024
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Jun 19, 2024
@lhotari lhotari requested a review from nlu90 June 20, 2024 05:34
@lhotari
Copy link
Member Author

lhotari commented Jun 20, 2024

There's also #22950 improving code and fixing bugs in this area.

@lhotari lhotari force-pushed the lh-producer-cache-expiration branch 3 times, most recently from 2992fb6 to 27d42b7 Compare June 20, 2024 13:33
@lhotari lhotari force-pushed the lh-producer-cache-expiration branch from 27d42b7 to 6b36081 Compare June 20, 2024 13:47
@lhotari lhotari merged commit 6fe8100 into apache:master Jun 25, 2024
47 of 51 checks passed
lhotari added a commit that referenced this pull request Jun 25, 2024
lhotari added a commit that referenced this pull request Jun 25, 2024
…onnectors (#22945)

(cherry picked from commit 6fe8100)
(cherry picked from commit f10708f)
lhotari added a commit that referenced this pull request Jun 25, 2024
lhotari added a commit that referenced this pull request Jun 25, 2024
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 27, 2024
…onnectors (apache#22945)

(cherry picked from commit 6fe8100)
(cherry picked from commit f10708f)
(cherry picked from commit 9f5432a)
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 27, 2024
…onnectors (apache#22945)

(cherry picked from commit 6fe8100)
(cherry picked from commit f10708f)
(cherry picked from commit 9f5432a)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Jul 1, 2024
…onnectors (apache#22945)

(cherry picked from commit 6fe8100)
(cherry picked from commit f10708f)
(cherry picked from commit 9f5432a)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants