fix: remove duplicated cell when interleave filter is applied #103
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pmakani
previously approved these changes
Apr 14, 2020
athakor
previously approved these changes
Apr 14, 2020
dmitry-fa
reviewed
Apr 14, 2020
| @@ -159,6 +162,9 @@ public void cellValue(ByteString newValue) { | |||
| * </ul> | |||
| * | |||
| * A flattened version of the {@link RowCell} map will be sorted correctly. | |||
| * | |||
| * <p>In case user applies {@link InterleaveFilter} than a {@link Cell} can appear more than | |||
| * once, but the duplicate cells will appear one after another. | |||
| */ | |||
There was a problem hiding this comment.
than -> then
it's not clear to me, what do you mean 'cells will appear one after another'
I would rephrase:
Applying {@link InterleaveFilter} may result in a row to contain duplicated cells.
There was a problem hiding this comment.
cells will appear one after another => I mean the duplicate cells will always be in group one after another (in that sequence)
There was a problem hiding this comment.
no 'but' here.
may be:
, but the duplicate cells will appear in a group i.e. one after another.
, where duplicates are grouped in sequences.
There was a problem hiding this comment.
Thanks for catching it 🙏
It seems Bigtable classic client & veneer client behaves differently for duplicated `cells`. A Row can contain duplicated cell when user applies `Interleave` filters. Behavior of both of these clients are: - Classic client: Here, we check if a `cell` contains some `labels` or not, If it does then we include that cell in end result, if it doesn't then we compare `timestamp` and `qualifier` with previous no label cell(Because a cell with label could have been produced by applying filters). - Veneer client: Here, we do not performs these checks.(Most of other client allows duplicate cells confirmed on `Go/NodeJs/C#` bigtable client. Not sure why but `python-bigtable` does not allows duplicate cells) **Assumption:** The labels are applied to determine which filters produced those cells, So we are including all cells where labels are present. chore: added javadoc to inform user about this bug - Added class JavaDoc and code comment for future reference. - reset the `previousNoLabelCell` in reset(). - added unit test to verify dedupe logic.
rahulKQL
force-pushed
the
updateRowResult
branch
from
9fa0895 to
84840ec
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It seems Bigtable classic client & veneer client behaves differently for duplicated
cells. A Row can contain duplicatedcellswhen the user appliesInterleavefilters. The behavior of both of these clients are:cellcontains somelabelsor not, If it does then we include that cell in end result if it doesn't then we comparetimestampandqualifierwith previous no label cell(Because a cell with alabelcould have been produced by applying filters).Go/NodeJs/C#bigtable client. Not sure why butpython-bigtabledoes not allow duplicate cells)Assumption: The labels are applied to determine which filters produced those cells, So we are including all cells where labels are present.