Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support BitSet and RunLength encoding lengths longer than Short.MAX_VALUE #37

Closed
astubbs opened this issue Nov 27, 2020 · 1 comment · Fixed by #42
Closed

Support BitSet and RunLength encoding lengths longer than Short.MAX_VALUE #37

astubbs opened this issue Nov 27, 2020 · 1 comment · Fixed by #42
Labels
enhancement New feature or request

Comments

@astubbs
Copy link
Contributor

astubbs commented Nov 27, 2020 •
edited
Loading

BitSet offset encodings are limited to Short.MAX_VALUE (32,000~ relative offsets) because a Short is used to serialise the Bitset run length. In order to support longer BitSets, need to use a bigger format like Integer (2,147,483,647 relative offsets).

RunLength has a similar problem - an integer is silently cast to a short, assuming a given run length is less than 32,000. Which may not always be the case. A new version of run length could use integer's instead. Could be another encoder that runs in parallel...
@astubbs astubbs mentioned this issue Nov 27, 2020
Closed
astubbs added a commit that referenced this issue Nov 27, 2020
@astubbs
bug: Revert 281f5fc fix: Turn off bitset length fatal check
0e0fb8f 
#35
RuntimeException when running with very high options in 0.2.0.0 (Bitset too long to encode) #35

This is due to Support BitSet encoding lengths longer than Short.MAX_VALUE #37.
Release 0.2.0.1 was put out incorrectly removing the check, but then it suddenly occurred to me what it was really checking.

However, this situation should not have arose because of the MAX values used in the test, there should never have been that many messages uncommitted anyway.
So two issues to fix here - first the reason there were so many to encode in the first place, and second upgrade the BitSet encoder to use Integers.

#37
Support BitSet encoding lengths longer than Short.MAX_VALUE #37
@astubbs astubbs added the enhancement New feature or request label Nov 27, 2020
@astubbs
Copy link
Contributor Author

astubbs commented Nov 27, 2020

Not as important if #38 is fixed

astubbs added a commit that referenced this issue Nov 27, 2020
@astubbs
bug: Revert 281f5fc fix: Turn off bitset length fatal check
d13df4c 
#35
RuntimeException when running with very high options in 0.2.0.0 (Bitset too long to encode) #35

This is due to Support BitSet encoding lengths longer than Short.MAX_VALUE #37.
Release 0.2.0.1 was put out incorrectly removing the check, but then it suddenly occurred to me what it was really checking.

However, this situation should not have arose because of the MAX values used in the test, there should never have been that many messages uncommitted anyway.
So two issues to fix here - first the reason there were so many to encode in the first place, and second upgrade the BitSet encoder to use Integers.

#37
Support BitSet encoding lengths longer than Short.MAX_VALUE #37
astubbs added a commit to astubbs/parallel-consumer that referenced this issue Nov 29, 2020
@astubbs
Gracefully drop bitset encoding if offset difference too large
225dc51 
RunLength encoding will still be used.

See confluentinc#37 Support BitSet encoding lengths longer than Short.MAX_VALUE
confluentinc#37

A test did try to cover this, but the offset difference wasn't large enough.
astubbs added a commit to astubbs/parallel-consumer that referenced this issue Nov 29, 2020
@astubbs
Gracefully drop bitset encoding if offset difference too large
d4cc21a 
RunLength encoding will still be used.

See confluentinc#37 Support BitSet encoding lengths longer than Short.MAX_VALUE
confluentinc#37

A test did try to cover this, but the offset difference wasn't large enough.
astubbs added a commit to astubbs/parallel-consumer that referenced this issue Nov 29, 2020
@astubbs
Gracefully drop bitset encoding if offset difference too large
b302aed 
RunLength encoding will still be used.

See confluentinc#37 Support BitSet encoding lengths longer than Short.MAX_VALUE
confluentinc#37

A test did try to cover this, but the offset difference wasn't large enough.
astubbs added a commit to astubbs/parallel-consumer that referenced this issue Nov 30, 2020
@astubbs
fix: Gracefully drop Bitset encoding if offset difference too large (…
d0993f1 
…short range)

RunLength encoding will still be used.

See confluentinc#37 Support BitSet encoding lengths longer than Short.MAX_VALUE
confluentinc#37

A test did try to cover this, but the offset difference wasn't large enough.
astubbs added a commit to astubbs/parallel-consumer that referenced this issue Nov 30, 2020
@astubbs
fix: Gracefully drop Bitset encoding if offset difference too large (…
5e0436b 
…short range)

RunLength encoding will still be used.

See confluentinc#37 Support BitSet encoding lengths longer than Short.MAX_VALUE
confluentinc#37

A test did try to cover this, but the offset difference wasn't large enough.
@astubbs astubbs mentioned this issue Nov 30, 2020
Merged
@astubbs astubbs changed the title Support BitSet encoding lengths longer than Short.MAX_VALUE Support BitSet and RunLength encoding lengths longer than Short.MAX_VALUE Nov 30, 2020
astubbs added a commit to astubbs/parallel-consumer that referenced this issue Nov 30, 2020
@astubbs
fix: Gracefully drop Bitset encoding if offset difference too large (…
64d2c1b 
…short range)

RunLength encoding will still be used.

See confluentinc#37 Support BitSet encoding lengths longer than Short.MAX_VALUE
confluentinc#37

A test did try to cover this, but the offset difference wasn't large enough.
astubbs added a commit to astubbs/parallel-consumer that referenced this issue Nov 30, 2020
@astubbs
fix: Gracefully drop Bitset encoding if offset difference too large (…
d7f5a7f 
…short range)

RunLength encoding will still be used.

See confluentinc#37 Support BitSet encoding lengths longer than Short.MAX_VALUE
confluentinc#37

A test did try to cover this, but the offset difference wasn't large enough.
astubbs added a commit that referenced this issue Nov 30, 2020
@astubbs
fix: Gracefully drop Bitset encoding if offset difference too large (…
f1e248f 
…short range)

RunLength encoding will still be used.

See #37 Support BitSet encoding lengths longer than Short.MAX_VALUE
#37

A test did try to cover this, but the offset difference wasn't large enough.
@astubbs astubbs mentioned this issue Dec 1, 2020
Merged
astubbs added a commit to astubbs/parallel-consumer that referenced this issue Dec 1, 2020
@astubbs
feature: confluentinc#37 Adds support for larger bitset and run-lengt…
1678e59 
…h encodings
astubbs added a commit that referenced this issue Dec 1, 2020
@astubbs
feature: #37 Adds support for larger bitset and run-length encodings
5dab32c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feature: Large bitset and run-length encodings astubbs/parallel-consumer
1 participant
@astubbs