[1105] Change Data Feed - PR 4 - UPDATE command #1146
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- add CDF logic into UpdateCommand.scala - add UpdateCDCSuite.scala GitOrigin-RevId: 97540779859d0c1f3a6207b68b5a929fdc030256
scottsand-db
commented
May 19, 2022
| tableName | ||
| .map(readDeltaTable(_)) | ||
| .getOrElse(readDeltaTableByPath(tempPath)) | ||
| .select("key", "value"), |
There was a problem hiding this comment.
some new tests write key, value, part (partition), yet we just want to assert on the key and value for the answer.
vkorukanti
approved these changes
May 24, 2022
| * @param target target we are updating into | ||
| * @param updateExpressions the update transformation to perform on the input DataFrame | ||
| * @param metrics transaction metrics | ||
| * @param dfWithEvaluatedCondition source DataFrame on which we will apply the update condition |
There was a problem hiding this comment.
isn't this DataFrame already has CONDITION_COLUMN_NAME? Why does the argument description say "on which we will apply the update condition with an additional column"?
There was a problem hiding this comment.
i'll change it to "on which we will apply the update expression". the evaluated condition is there (true or false), but the actual update on the row has not been performed.
| target, | ||
| updateExpressions, | ||
| condition, | ||
| targetDf.withColumn(CONDITION_COLUMN_NAME, new Column(condition)), |
There was a problem hiding this comment.
We are adding the new column here but dropping the column in withUpdatedColumns. Is it cleaner to just pass the targetDf and let the withUpdatedColumns build on the targetDf?
jbguerraz
pushed a commit
to jbguerraz/delta
that referenced
this pull request
Jul 6, 2022
See the project plan at delta-io#1105. This PR adds CDF to the UPDATE command, during which we generate both preimage and postimage CDF data. This PR also adds UpdateCDCSuite which adds basic tests for these CDF changes. As a high-level overview of how this CDF-update operation is performed, when we find a row that satisfies the update condition, we `explode` an array containing the pre-image, post-image, and main-table updated rows. The pre-image and post-image rows are appropriately typed with the corresponding CDF_TYPE, and the main-table updated row has CDF_TYPE `null`. Thus, the first two rows will be written to the cdf parquet file, with the latter is written to standard main-table data parquet file. Closes delta-io#1146 GitOrigin-RevId: 47413c5345bb97c0e1303a7f4d4d06b89c35ab7a
jbguerraz
pushed a commit
to jbguerraz/delta
that referenced
this pull request
Jul 6, 2022
See the project plan at delta-io#1105. This PR adds CDF to the UPDATE command, during which we generate both preimage and postimage CDF data. This PR also adds UpdateCDCSuite which adds basic tests for these CDF changes. As a high-level overview of how this CDF-update operation is performed, when we find a row that satisfies the update condition, we `explode` an array containing the pre-image, post-image, and main-table updated rows. The pre-image and post-image rows are appropriately typed with the corresponding CDF_TYPE, and the main-table updated row has CDF_TYPE `null`. Thus, the first two rows will be written to the cdf parquet file, with the latter is written to standard main-table data parquet file. Closes delta-io#1146 GitOrigin-RevId: 47413c5345bb97c0e1303a7f4d4d06b89c35ab7a
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See the project plan at #1105.
This PR adds CDF to the UPDATE command, during which we generate both preimage and postimage CDF data.
This PR also adds UpdateCDCSuite which adds basic tests for these CDF changes.
As a high-level overview of how this CDF-update operation is performed, when we find a row that satisfies the update condition, we
explodean array containing the pre-image, post-image, and main-table updated rows.The pre-image and post-image rows are appropriately typed with the corresponding CDF_TYPE, and the main-table updated row has CDF_TYPE
null. Thus, the first two rows will be written to the cdf parquet file, with the latter is written to standard main-table data parquet file.