Add and use CLPMutableForwardIndexV2 by default to improve ingestion performance and efficiency #14241
Conversation
| protected int _nextDocId = 0; | ||
| protected int _nextDictVarDocId = 0; | ||
| protected int _nextEncodedVarId = 0; | ||
| protected int _bytesRawFwdIndexDocIdStartOffset = Integer.MAX_VALUE; |
There was a problem hiding this comment.
is this only for CLPV2? can we add a comment on how this value is dynamic updated?
There was a problem hiding this comment.
Yes, this is only used for CLPV2 (but might be used for CLPV3 in the future). Internally, we have classes which extends the functionality of this class, so some class member variables are marked as protected.
There was a problem hiding this comment.
The value is updated at the end of appendEncodedMessage method call. I will add a comment to make this clear.
...main/java/org/apache/pinot/segment/local/realtime/impl/forward/CLPMutableForwardIndexV2.java
Outdated
Show resolved
Hide resolved
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #14241 +/- ##
============================================
+ Coverage 61.75% 63.84% +2.09%
- Complexity 207 1535 +1328
============================================
Files 2436 2627 +191
Lines 133233 144845 +11612
Branches 20636 22162 +1526
============================================
+ Hits 82274 92477 +10203
- Misses 44911 45526 +615
- Partials 6048 6842 +794
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
…rdIndex during mutable->immutable segment conversion.
jackluo923
changed the title
…performance and efficiency apache#14241
Sure, I will make the change in another PR. |
Summary
CLPMutableForwardIndexV2is a comprehensive improvement overCLPMutableForwardIndexV1, offering enhancements in both performance and functionality.CLPMutableForwardIndexV2is also compatible withCLPMutableForwardIndexandCLPMutableForwardIndexV2is now the default CLP mutable forward index to be used within Pinot.This PR is the first in a series of three PRs:
Compatibility
CLPMutableForwardIndexV2is compatible withCLPMutableForwardIndex. A unit test validating this compatibility can be found inCLPMutableForwardIndexTest#testStringV2(). In the future, we may consider replacingCLPMutableForwardIndexwithCLPMutableForwardIndexV2. However, keeping both implementations for now provides a safety net, allowing us to revert to the older version should any unforeseen issues arise with V2.Details
Performance Improvement
The serialization/deserialization overhead is significantly lower compared to
CLPMutableForwardIndexV1. With the upgrade toclp-ffi-javaversion0.4.7, we can now directly access the raw byte[] content of CLP-encoded messages, rather than operating over boxed String objects for logtype and dictVar and Long objects for encodedVar. On the forward index storage side, the composite mutable forward index now exclusively stores primitive types. For decoding, we also work with raw byte and long arrays, eliminating the need to first encode the bytes into a String/Long before passing them to CLP's decoder to unbox and decode the log message.Functionality Improvement
The new mutable forward index implements a composite index for string-typed columns with dynamic encoding options:
Initially, CLP encoding transforms an extremely high-cardinality log message string into three data columns:
The logtype and dictionary variables are dictionary-encoded, while the encoded variables are stored as longs. Notably, both
encodedVarIdsandencodedVarsare multi-valued, but they are stored using a flattened single-value mutable forward index, along with a separate forward index to capture the end offsets for each multi-valued document. This approach is necessary because the maximum number of values per document is unknown during ingestion, unlike the existing multi-value forward index, which requires this information upfront. During the conversion from mutable to immutable forward index, the two single-value mutable indices are merged into a single immutable multi-valued forward index, as the max length is now known post ingestion (conversion time).During ingestion, if the cardinality of either the
logtypeIdordictVarIDexceeds a predefined threshold, the ingestion mode switches to a raw bytes forward index for subsequent documents. Maintaining very large dictionaries is inefficient in Pinot due to memory and I/O constraints (memory-mapped). Switching to a raw bytes forward index helps avoid these issues. During reads, if the requesteddocIdis in the raw forward index, the raw bytes are returned. Otherwise, the log type, dictionary variables, and encoded variables are decoded using the CLP decoder to return the original log message's bytes.Notes:
Writes are strictly sequential, while reads can be performed randomly. The supported append operations are:
setString(int docId, String value)- Encodes the log message using CLP and invokesappendEncodedMessage(@NotNull EncodedMessage clpEncodedMessage)appendEncodedMessage(@NotNull EncodedMessage clpEncodedMessage)Future Work: