Make RecordEnricher extend RecordTransformer #14601
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Jackie-Jiang
added
incompatible
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #14601 +/- ##
============================================
+ Coverage 61.75% 63.96% +2.21%
- Complexity 207 1572 +1365
============================================
Files 2436 2687 +251
Lines 133233 147650 +14417
Branches 20636 22629 +1993
============================================
+ Hits 82274 94442 +12168
- Misses 44911 46270 +1359
- Partials 6048 6938 +890
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
Jackie-Jiang
force-pushed
the
make_enricher_transformer
branch
from
09c5d51 to
cc9e2a4
Compare
...cal/src/main/java/org/apache/pinot/segment/local/recordtransformer/CompositeTransformer.java
Show resolved
Hide resolved
pinot-spi/src/main/java/org/apache/pinot/spi/recordtransformer/enricher/RecordEnricher.java
Show resolved
Hide resolved
swaminathanmanish
approved these changes
Dec 6, 2024
| @@ -141,6 +141,7 @@ private void resetBuffer() | |||
| @Override | |||
| public void collect(GenericRow row) | |||
| throws IOException { | |||
| // TODO: Revisit whether we should transform the row | |||
There was a problem hiding this comment.
Can you explain what the TODO means?
There was a problem hiding this comment.
In flush() we will use the collected row to create a segment, which applies the transforms again
This was referenced Dec 6, 2024
davecromberge
pushed a commit
to davecromberge/pinot
that referenced
this pull request
Dec 10, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Address #13049
Another attempt after #13086 and #13704
RecordEnricherextendRecordTransformerRecordEnricherPipelineintoCompositeTransformerRecordTransformer.getInputColumns()to return the required input columns, and simplifyIngestionUtils.getFieldsForRecordExtractor()Incompatible
RecordTransformeris moved topinot-spimodule:org.apache.pinot.spi.recordtransformerRecordEnricheris moved fromorg.apache.pinot.spi.recordenrichertoorg.apache.pinot.spi.recordtransformer.enricher