Store bitmap sizes along with bitmaps in the segment format

[leventov]

Absence of precomputed bitmap sizes makes implementation of features like #3878 and #3848 much less effective than it could be.

[gianm]

One thing to be careful of here is that 4 bytes per bitmap can add up quickly for high cardinality dimensions. Still a useful idea though.

[leventov]

@gianm we can use condensed integer encodings, where the highest bit of each byte is a continuation flag. Numbers 0-127 take just 1 byte.

[gianm]

yes, for sure that is a valid approach. Just wanting to make sure that this issue is considered. It may also be worth while to skip the size field for small bitmaps where computing size is fast. Odds are, high cardinality dimensions will generally have small bitmaps. I'm not sure though since I'm not sure of the speed difference between reading a vsize int and computing size for a small bitmap.

[github-actions]

This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

[github-actions]

This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.