0.5.3.2

Fixes

• fix: include inflight message count in polling backpressure logic (#836)
• fix: message loss on closing or partitions revoked (#827) fixes (#826)
• fix: unbounded retry queue growth preventing polling from being throttled and leading to OOM (#834) fixes (#832, #817)

Note

#836 introduces a change in how buffer size is calculated as now inflight messages are counted as part of buffer size - so behaviour of existing applications may change and pausing of consumer happen sooner.

GH Generated

What's Changed

• fix for message loss for issue 826 by @sangreal in #827
• chore: update repo by service bot by @ConfluentJenkins in #813
• Fix retry queue implementation to prevent unbounded growth by @rkolesnev in #834
• Fix buffer size include in flight messages in backpressure calculation logic by @rkolesnev in #836

Full Changelog: 0.5.3.1...0.5.3.2

0.5.3.1

Fixes

• fix: ConcurrentModificationException Happened while high load and draining (#822) fixes (#821)
• fix: safely completing doClose() (#818) partially fixes (#809)
• Improved offset commit retry. Add support for SaslAuthenticationException retry timeout (#819), partially fixes (#809) in Commit_Sync mode

GH Generated

What's Changed

• remove publish auto-promotion by @rkolesnev in #807
• fix: safely completing doClose() by @gtassone in #818
• switch retryQueue to ConcurrentSkipListSet to avoid ConcurrentModific… by @sangreal in #822
• Improved offset commit retry. Add support for SaslAuthenticationException retry timeout by @wushilin in #819
• Update changelog for release by @skl83 in #823
• Update README for release by @skl83 in #824
• Revert release 0.5.3.1 and update Sonatype creds to use token by @skl83 in #828

New Contributors

• @gtassone made their first contribution in #818
• @wushilin made their first contribution in #819
• @skl83 made their first contribution in #823

Full Changelog: 0.5.3.0...0.5.3.1

0.5.3.0

Fixes

• fix: ReactorProcessor - run used-defined function in provided scheduler rather than in pc-pool thread (#798 / #794), fixes (#793)
• fix: fix issue for cannot close and exit properly when re-balancing storm (#787)
• fix: Support for PCRetriableException in ReactorProcessor (#733)
• fix: NullPointerException on partitions revoked (#757)
• fix: remove lingeringOnCommitWouldBeBeneficial and unused imports (#732)
• fix: Fix failing auto-commit check for kafka-clients >= v3.7.0 (#721)
• fix: Fix redundant rebalance callback in LongPollingMockConsumer for Kafka >= 3.6 (#765)

Improvements

• improvement: stale containers exclusion and handling improvement (#779)
• improvement: add multiple caches for accelerating available container count calculation （#667）
• improvement: RecordContext now exposes lastFailureReason (#725)

Notable dependency upgrades

• build(deps): Bump Kafka to 3.7.0

GH Generated

What's Changed

• add multiple caches for accelerating available container count calculation by @sangreal in #667
• GH-725: RecordContext should provide access to lastFailureReason by @jvissers in #727
• Fix the example code for skipping records to work properly by @seungy0 in #719
• Enable semaphore as part of migrating builds from jenkins to semaphoreCI by @rkolesnev in #736
• Enable semaphore by @rkolesnev in #737
• chore: update repo by service bot by @ConfluentJenkins in #738
• setting up semaphore CI job(s) by @rkolesnev in #739
• chore: update repo by service bot by @ConfluentJenkins in #741
• fix: Support for PCRetriableException in ReactorProcessor (#733) by @brp-jakob in #734
• chore: update repo by service bot by @ConfluentJenkins in #742
• update semaphore to unshallow git for license check to work correctly by @rkolesnev in #745
• chore: update repo by service bot by @ConfluentJenkins in #744
• remove make / makefile usage by @rkolesnev in #749
• Fix/public ci job by @rkolesnev in #759
• Remove Jenkinsfile as Jenkins CI jobs are migrated to Semaphore by @rkolesnev in #760
• remove lingeringOnCommitWouldBeBeneficial and unused imports by @sangreal in #732
• fix: NullPointerException on partitions revoked (#757) by @sangreal in #758
• Fix redundant rebalance callback in LongPollingMockConsumer for Kafka >= 3.6 by @nscuro in #765
• Fix failing auto-commit check for kafka-clients >= v3.7.0 by @nscuro in #762
• Remove redundant line in the React example by @maksmas in #764
• build(deps): bump com.google.guava:guava from 32.1.1-jre to 33.2.0-jre by @dependabot in #761
• build(deps): bump junit.version from 5.10.1 to 5.10.2 by @dependabot in #768
• build(deps): bump org.slf4j:slf4j-api from 2.0.7 to 2.0.13 by @dependabot in #755
• docs: readme - move changes in #735 into template by @rkolesnev in #767
• build(deps-dev): bump junit.platform.version from 1.10.1 to 1.10.2 by @dependabot in #771
• build(deps): bump org.apache.maven.plugins:maven-deploy-plugin from 3.1.1 to 3.1.2 by @dependabot in #770
• Bump mockito.version from 5.10.0 to 5.12.0 by @dependabot in #772
• Bump io.micrometer:micrometer-core from 1.12.2 to 1.13.0 by @dependabot in #773
• build(deps): bump testcontainers.version from 1.19.3 to 1.19.8 by @dependabot in #769
• Update CODEOWNERS by @johnbyrnejb in #784
• add release pipeline for semaphore by @rkolesnev in #783
• update semaphore release related commands by @rkolesnev in #788
• Move tasks block into semaphore block in service.yml by @rkolesnev in #791
• fix issue #787 for cannot close and exit properly when rebalancing storm by @sangreal in #789
• chore: update repo by service bot by @ConfluentJenkins in #792
• stale containers exclusion and handling improvement by @sangreal in #779
• Add wait cycles in a test to help de-flake it. by @rkolesnev in #795
• Run used-defined function in provided scheduler rather than in pc-pool thread by @rkolesnev in #798
• change version in changelog / readme for 0.5.3.0 release by @rkolesnev in #799
• push changes on release prepare steps by @rkolesnev in #800
• dont do release:perform in release task - its part of deploy by @rkolesnev in #801
• Fix: update vault secret and revert release 0.5.3.0 prep commits by @rkolesnev in #802
• Backout release add ossrh handling by @rkolesnev in #805
• Revert release update script path by @rkolesnev in #806

New Contributors

• @jvissers made their first contribution in #727
• @seungy0 made their first contribution in #719
• @ConfluentJenkins made their first contribution in #738
• @brp-jakob made their first contribution in #734
• @nscuro made their first contribution in #765
• @maksmas made their first contribution in #764
• @johnbyrnejb made their first contribution in #784

Full Changelog: 0.5.2.8...0.5.3.0

0.5.2.8

Fixes

• fix: Fix equality and hash code for ShardKey with array key (#638), resolves (#579)
• fix: Fix target loading computation for inflight records (#662)
• fix: Fix synchronisation logic for transactional producer commit affecting non-transactional usage (#665), resolves (#637)
• fix: Fix for race condition in partition state clean/dirty tracking (#666), resolves (#664)

Improvements

• feature: Make PC message buffer size configurable - two new configuration options for controlling buffer size added (#682)

GH generated

• Fix key equality by @rkolesnev in #638
• Remove duplicated batchSize factor from target load; add a test case by @cloudhunter89 in #662
• Fix synchronisation logic between commits by @rkolesnev in #665
• Fix for race condition in partition state clean/dirty tracking by @rkolesnev in #666
• use https for repositories by @rkolesnev in #683
• Make PC message buffer size configurable by @rkolesnev in #682

Dependency updates omitted for readability

New Contributors

• @cloudhunter89 made their first contribution in #662

Full Changelog: 0.5.2.7...0.5.2.8

0.5.2.7

Fixes

• fix: Return cached pausedPartitionSet (#620), resolves (#618)
• fix: Parallel consumer stops processing data sometimes (#623), fixes (#606)
• fix: Add synchronization to ensure proper intializaiton and closing of PCMetrics singleton (#627), fixes (#617)
• fix: Readme - metrics example correction (#614)
• fix: Remove micrometer-atlas dependency (#628), fixes (#625)

Improvements

• Refactored metrics implementation to not use singleton - improves meter separation, allows correct metrics subsystem operation when multiple parallel consumer instances are running in same java process (#630), fixes (#617) improves on (#627)

GH generated

• fix : for issue 606 : exclude stale containers from counting and add remove add remove staled work containers function in ProcessingShard by @sangreal in #623
• fix: Return cached pausedPartitionSet by @acktsap in #620
• PL-464 remove micrometer-atlas dependency by @rkolesnev in #628
• minor: Change Supplier -> SupplierUtils for consistency by @acktsap in #619
• add synchronization to ensure proper intializaiton and closing of PCM… by @antonmos in #627
• fix readme doc by @flashmouse in #624
• Fix/refactor metrics singleton by @rkolesnev in #630
• build(deps): bump io.projectreactor:reactor-core from 3.5.7 to 3.5.9 by @dependabot in #626
• build(deps): bump org.xerial.snappy:snappy-java from 1.1.10.1 to 1.1.10.3 by @dependabot in #610
• build(deps-dev): bump junit.platform.version from 1.10.0-RC1 to 1.10.0 by @dependabot in #612
• build(deps): bump maven-site-plugin from 4.0.0-M8 to 4.0.0-M9 by @dependabot in #603

New Contributors

• @sangreal made their first contribution in #623
• @antonmos made their first contribution in #627

Full Changelog: 0.5.2.6...0.5.2.7

0.5.2.6

Improvements

• feature: Micrometer metrics (#594)

Fixes

• fix: Don’t drain mode shutdown kills inflight threads (#559)
• fix: Drain mode shutdown doesn’t pause consumption correctly (#552)
• fix: RunLength offset decoding returns 0 base offset after no-progress commit - related to (#546)
• fix: Transactional PConsumer stuck while rebalancing - related to (#541)

GH generated:

What's Changed

• fix: runlength decoding after no-progress commit by @rkolesnev in #563
• Fix deadlock between pc-control and pc-broker-poll threads where partitions are revoked by @nachomdo in #548
• build(deps-dev): bump mockito.version from 4.9.0 to 5.1.1 by @dependabot in #556
• build(deps): bump vertx.version from 4.3.6 to 4.4.1 by @dependabot in #566
• build(deps): bump versions-maven-plugin from 2.14.1 to 2.15.0 by @dependabot in #555
• build(deps-dev): bump threeten-extra from 1.7.1 to 1.7.2 by @dependabot in #528
• Do not create trace log message when trace logging is not enabled by @lennehendrickx in #544
• build(deps): bump maven-release-plugin from 3.0.0-M7 to 3.0.1 by @dependabot in #580
• PL-211: Update dependencies from dependabot, Add mvnw, use mvnw in jenkins by @eddyv in #583
• PL-211: update dependencies from dependabot by @eddyv in #589
• add instance id into thread name by @flashmouse in #578
• workerThreadPool is initialized lazily by @pweso in #531
• build(deps): bump reactor-core from 3.5.1 to 3.5.7 by @dependabot in #585
• Pl 175 DRAIN mode fix by @niamhthornbury in #591
• build(deps): bump junit.version from 5.10.0-M1 to 5.10.0-RC1 by @dependabot in #598
• build(deps): bump guava from 32.0.1-jre to 32.1.1-jre by @dependabot in #595
• build(deps): bump vertx.version from 4.4.3 to 4.4.4 by @dependabot in #590
• build(deps-dev): bump junit.platform.version from 1.10.0-M1 to 1.10.0-RC1 by @dependabot in #599
• improves: Error in onPartitionsAssigned in parallel consumer by @nachomdo in #537
• Feature/health metrics micrometer by @rkolesnev in #613
• fix: example in readme, license headers by @rkolesnev in #614
• PL-176 handle close dont drain mode gracefully by @rkolesnev in #615

New Contributors

• @lennehendrickx made their first contribution in #544
• @eddyv made their first contribution in #583
• @flashmouse made their first contribution in #578
• @pweso made their first contribution in #531
• @niamhthornbury made their first contribution in #591

Full Changelog: 0.5.2.5...0.5.2.6

0.5.2.5

Improvements

• perf: Adds a caching layer to work management to alleviate O(n) counting (#530)

Fixes

• fixes: #195 NoSuchFieldException when using consumer inherited from KafkaConsumer (#469)
• fix: After new performance fix PR#530 merges - corner case could cause out of order processing (#534)
• fix: Cleanup WorkManager's count of in-progress work, when work is stale after partition revocation (#547)

GH generated:

What's Changed

• fixes: #195 NoSuchFieldException when using consumer inherited from KafkaConsumer by @astubbs in #469
• tests: Demonstrates how to use MockConsumer with PC for issue #176 by @astubbs in #470
• logs: Add partition to warning upon log Truncation by @Ehud-Lev-Forter in #489
• minor: fixes #486: Missing generics in JStreamParallelStreamProcessor by @astubbs in #491
• build(deps): wiremock, mvn-release, reactor, podam, postgresql, junit-pioneer by @astubbs in #495
• minor: tests: Removes empty/not implemented tests by @astubbs in #493
• tests: Extra test coverage for when there is less keys than threads by @astubbs in #450
• docs: Remove legacy config maxUncommittedMessagesToHandle from readme by @astubbs in #503
• build(deps): vertx, maven-dependency, testcontainers, logback, mockito by @astubbs in #504
• docs: Changelog update by @astubbs in #507
• minor: JStream review / generics fixes by @astubbs in #505
• build: Update JenkinsFile to match upstream changes by @astubbs in #521
• minor: Remove dangling API methods which were kept by accident by @astubbs in #522
• minor: Fix enum naming, tests tweak by @astubbs in #517
• build(deps): bump vertx.version, maven-install, slf4j-api, reactor-core, versions-maven-plugin by @astubbs in #523
• fixes #327: Improve error message when Kafka Streams magic number found by @nachomdo in #482
• Add missing @UtilityClass to utils class by @acktsap in #518
• docs: Yourkit Attribution by @astubbs in #529
• fix: After new performance fix PR#530 merges - corner case could cause out of order processing by @astubbs in #534
• Fixes #547 - Cleanup WorkManager#numberRecordsOutForProcessing when w… by @RaviKalasapur in #549

New Contributors

• @Ehud-Lev-Forter made their first contribution in #489
• @nachomdo made their first contribution in #482
• @acktsap made their first contribution in #518
• @RaviKalasapur made their first contribution in #549

Full Changelog: 0.5.2.4...0.5.2.5

0.5.2.4

Improvements

• feature: Simple PCRetriableException to remove error spam from logs (#444)

Fixes

• fixes #409: Adds support for compacted topics and commit offset resetting (#425)
  • Truncate the offset state when bootstrap polled offset higher or lower than committed
  • Prune missing records from the tracked incomplete offset state, when they're missing from polled batches
• fix: Improvements to encoding ranges (int vs long) #439
  • Replace integer offset references with long - use Long everywhere we deal with offsets, and where we truncate down, do it exactly, detect and handle truncation issues.

GH Generated:

What's Changed

• build: Fix maven scm tag to HEAD by @astubbs in #434
• fixes #409: Adds support for compacted topics and commit offset resetting by @astubbs in #425
• build(deps): bump vertx.version from 4.3.3 to 4.3.4 by @dependabot in #427
• build(deps): bump maven-jar-plugin from 3.2.2 to 3.3.0 by @dependabot in #426
• build(deps): bump testcontainers.version from 1.17.3 to 1.17.5 by @dependabot in #438
• docs: Various javadoc and @StandardException by @astubbs in #343
• build:(deps) Bump JUnit, Reactor by @astubbs in #447
• build:(deps) Bump Slf4j, Logback by @astubbs in #448
• build(deps): bump jabel from 0.4.2 to 1.0.0 (Java 19 support) by @astubbs in #449
• refactor #409: Clarify truncation code by @astubbs in #445
• fixes: NoSuchElementException in LoopingResumingIterator by @HiremathPanchaxari in #435
• feature: Simple retriable exception to remove error spam from logs by @astubbs in #444
• build(deps-dev): Fix Slf4j setup caused by shaded Wiremock dep (slf4j/logback collisions) by @astubbs in #451
• Update service.yml - disable semaphore by @vinnybod in #458
• fix: Improvements to encoding ranges (int vs long) by @astubbs in #439
• refactor: SortedSet's all the way down (Set->SortedSet) by @astubbs in #463
• Fixes/test improvements by @astubbs in #465
• docs: Add Reactor example and clarify bulk transaction system by @astubbs in #452

New Contributors

• @HiremathPanchaxari made their first contribution in #435
• @vinnybod made their first contribution in #458

Full Changelog: 0.5.2.3...0.5.2.4

0.5.2.3

Improvements

• Transactional commit mode system improvements and docs (#355)

  • Clarifies transaction system with much better documentation.
  • Fixes a potential race condition which could cause offset leaks between transactions boundaries.
  • Introduces lock acquisition timeouts.
  • Fixes a potential issue with removing records from the retry queue incorrectly, by having an inconsistency between compareTo and equals in the retry TreeMap.

• Adds a very simple Dependency Injection system modeled on Dagger (#398)

• Various refactorings e.g. new ProducerWrap

• Dependencies

  • build(deps): prod: zstd, reactor, dev: podam, progressbar, postgresql maven-plugins: versions, help (#420)
  • build(deps-dev): bump postgresql from 42.4.1 to 42.5.0
  • bump podam, progressbar, zstd, reactor
  • build(deps): bump versions-maven-plugin from 2.11.0 to 2.12.0
  • build(deps): bump maven-help-plugin from 3.2.0 to 3.3.0
  • build(deps-dev): bump Confluent Platform Kafka Broker to 7.2.2 (#421)
  • build(deps): Upgrade to AK 3.3.0 (#309)

Fixes

• fixes #419: NoSuchElementException during race condition in PartitionState (#422)
• Fixes #412: ClassCastException with retryDelayProvider (#417)
• fixes ShardManager retryQueue ordering and set issues due to poor Comparator implementation (#423)

GH generated:

What's Changed

• Improvements/DI module by @astubbs in #398
• Improvements/transaction docs large refactorings #399 - Fix for merge error by @astubbs in #406
• Transaction system improvements and docs by @astubbs in #355
• build(deps): Upgrade to AK 3.3.0 by @astubbs in #309
• build(deps-dev): bump Confluent Platform Kafka Broker to 7.2.2 by @astubbs in #421
• build(deps): prod: zstd, reactor, dev: podam, progressbar, postgresql maven-plugins: versions, help by @astubbs in #420
• Fixes #412: ClassCastException with retryDelayProvider by @astubbs in #417
• ide: Add in ignored bisect helper and run targets by @astubbs in #347
• fixes #419 NoSuchElementException during race condition in PartitionState by @astubbs in #422
• docs: Clarify offset storage behaviour by @astubbs in #424
• fixes #327: Friendlier error for encoding problems & Exception name normalisation by @astubbs in #354
• fixes ShardManager retryQueue ordering and set issues due to poor Comparator implementation by @astubbs in #423

Full Changelog: 0.5.2.2...0.5.2.3

0.5.2.2

What's Changed

• fixes #375: Mockito dependency should be test scoped by @wonmo01 in #376

Dependencies

• build(deps-dev): bump postgresql, install, info, unit, deploy, resource by @astubbs in #370
• build(deps): bump reactor, mockito, junit-platorm, site, threeten (#386)
• build(deps): bump vertx.version from 4.3.2 to 4.3.3 by @dependabot in #387
• build(deps): bump maven-javadoc-plugin from 3.4.0 to 3.4.1 by @dependabot in #388

New Contributors

• @wonmo01 made their first contribution in #376

Full Changelog: 0.5.2.1...0.5.2.2