Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tests: Extra test coverage for when there is less keys than threads #450

Merged
merged 149 commits into from
Dec 7, 2022

Conversation

astubbs
Copy link
Contributor

@astubbs astubbs commented Oct 20, 2022

Extra test coverage from some discussion in #433

dependabot bot and others added 30 commits May 20, 2022 16:33
Bumps `testcontainers.version` from 1.17.1 to 1.17.2.

Updates `testcontainers` from 1.17.1 to 1.17.2
- [Release notes](https://github.com/testcontainers/testcontainers-java/releases)
- [Changelog](https://github.com/testcontainers/testcontainers-java/blob/master/CHANGELOG.md)
- [Commits](testcontainers/testcontainers-java@1.17.1...1.17.2)

Updates `kafka` from 1.17.1 to 1.17.2
- [Release notes](https://github.com/testcontainers/testcontainers-java/releases)
- [Changelog](https://github.com/testcontainers/testcontainers-java/blob/master/CHANGELOG.md)
- [Commits](testcontainers/testcontainers-java@1.17.1...1.17.2)

Updates `junit-jupiter` from 1.17.1 to 1.17.2
- [Release notes](https://github.com/testcontainers/testcontainers-java/releases)
- [Changelog](https://github.com/testcontainers/testcontainers-java/blob/master/CHANGELOG.md)
- [Commits](testcontainers/testcontainers-java@1.17.1...1.17.2)

Updates `postgresql` from 1.17.1 to 1.17.2
- [Release notes](https://github.com/testcontainers/testcontainers-java/releases)
- [Changelog](https://github.com/testcontainers/testcontainers-java/blob/master/CHANGELOG.md)
- [Commits](testcontainers/testcontainers-java@1.17.1...1.17.2)

---
updated-dependencies:
- dependency-name: org.testcontainers:testcontainers
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: org.testcontainers:kafka
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: org.testcontainers:junit-jupiter
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: org.testcontainers:postgresql
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [versions-maven-plugin](https://github.com/mojohaus/versions-maven-plugin) from 2.10.0 to 2.11.0.
- [Release notes](https://github.com/mojohaus/versions-maven-plugin/releases)
- [Changelog](https://github.com/mojohaus/versions-maven-plugin/blob/master/ReleaseNotes.md)
- [Commits](mojohaus/versions@versions-maven-plugin-2.10.0...versions-maven-plugin-2.11.0)

---
updated-dependencies:
- dependency-name: org.codehaus.mojo:versions-maven-plugin
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps `vertx.version` from 4.3.0 to 4.3.1.

Updates `vertx-web-client` from 4.3.0 to 4.3.1

Updates `vertx-junit5` from 4.3.0 to 4.3.1

---
updated-dependencies:
- dependency-name: io.vertx:vertx-web-client
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: io.vertx:vertx-junit5
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [postgresql](https://github.com/pgjdbc/pgjdbc) from 42.3.5 to 42.4.0.
- [Release notes](https://github.com/pgjdbc/pgjdbc/releases)
- [Changelog](https://github.com/pgjdbc/pgjdbc/blob/master/CHANGELOG.md)
- [Commits](pgjdbc/pgjdbc@REL42.3.5...REL42.4.0)

---
updated-dependencies:
- dependency-name: org.postgresql:postgresql
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
- CVE-2020-8908: Files::createTempDir local information disclosure vulnerability #4011
- Only used transitively from tests, and is a deprecated function
- google/guava#4011
…rs.version-1.17.2' into improvements/version-bumps
…ql-postgresql-42.4.0' into improvements/version-bumps
….mojo-versions-maven-plugin-2.11.0' into improvements/version-bumps
New issues with WireMocks dep on Jetty BOM ~v9
New issues with WireMocks dep on Jetty BOM ~v9
…that

Especially given we also use dependabot. If anyone is concerned to that level, they can use the plugin in their end user applications.
…that

Especially given we also use dependabot. If anyone is concerned to that level, they can use the plugin in their end user applications.
@astubbs astubbs changed the title START: experimental test for less keys than threads Experimental test for less keys than threads Oct 25, 2022
@what-the-diff
Copy link

what-the-diff bot commented Nov 21, 2022

  • The range of offsets is now encoded as a long, rather than an int.
  • A new utility class was added to support the conversion from Long to Short (MathUtils)
  • Range has been updated with some additional methods for testing purposes and also uses MathUtils internally when converting between types
  • BitSetEncoder has been refactored so that it can encode ranges up to Integer#MAX_VALUE in size, which should be sufficient for all practical use cases
  • Changed the spelling of deserialisation to deserialization
  • Added a version field for encoding versions and updated constructor accordingly
  • Updated log messages in register method with size instead of site (which was not defined)
  • Added a new encoder: ByteBufferEncoder
  • Changed the type of activeEncoders from Set to ConcurrentHashMap.KeySetView<OffsetEncoder, Boolean> so that we can remove an encoder while traversing it (in invoke())
  • Removed BitSet encoding for v1 as it is not supported by Java 11+ and added some logging around this change
  • Fixed bug where lengthBetweenBaseAndHighOffset was being calculated incorrectly when highestSucceededOffset == KAFKA_OFFSET_ABSENCE (-1) which caused negative lengths in RunLengthCompressed encoding
  • Renamed RunlengthV1EncodingNotSupported to RunLengthV1EncodingNotSupported
  • Added a new class called RunLengthV2EncodingNotSupported which is thrown when the run-length encoding for version 2 (Integer) fails due to being too big
  • Changed some loops from using Range#range(int) to use Range#range(long). This was done because we were getting overflows in our tests, and this change fixes that issue by making sure all offsets are longs instead of ints.
  • Fixed an issue where incompleteOffsetsBelowHighestSucceeded() would return null if there were no completed offsets yet - it now returns empty set as expected behaviour should be (and what's documented in javadoc). Also changed Set -> SortedSet. The reason why I did this is so that we can easily get the first element without having to do any sort of conversion or casting etc... It also makes sense since they're sorted anyway! :)
  • Removed unused import io/confluent/csid/utils/Range; from OffsetSimultaneousDecoderTestBase and added back into GeneralIntegrationTests after removing usage of range(). See point 3 above for more details on why I removed usages of range().
  • Change the range() method to Range.range().
  • Add a new test case for lessKeysThanThreads(), which is used to check whether there are different computational results obtained with different max concurrency configurations for the same parallel consumer Different computational results obtained with different max concurrency configurations for the same parallel consumer #433, and add some log information in this test case.
  • Fix bug: when we have more keys than records, then we'll have a sequence size of 1, so round up (see build: Fix maven scm tag to HEAD #434).
  • Added imports
  • Changed -1 to KAFKA_OFFSET_ABSENCE in RunLengthEncoderTest class
  • Removed DRY code from noGaps() and noGapsSerialisation() methods of RunLengthEncoderTest class
  • Replaced Assertions with Truth assertions for vTwoIntegerOverflow method of RunLengthEncoderTest class
  • Refactored testRunLength method into a private static one, added TypeKind enum as parameter and changed the error assertion messages accordingly in order to make it more readable

…threads

# Conflicts:
#	parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/offsets/OffsetBitSet.java
@astubbs astubbs changed the title Experimental test for less keys than threads tests: Extra test coverage for when there is less keys than threads Dec 7, 2022
@astubbs astubbs marked this pull request as ready for review December 7, 2022 14:45
@astubbs astubbs merged commit 78aab49 into confluentinc:master Dec 7, 2022
@astubbs astubbs deleted the exp/less-keys-than-threads branch December 7, 2022 15:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant