[1105] CDF + Vacuum tests #1177
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
GitOrigin-RevId: 130fa5e5e53bb62b0e58b01b1cd188ec662627cb
mmengarelli
pushed a commit
to mmengarelli/delta.io
that referenced
this pull request
Jun 8, 2022
GitOrigin-RevId: ceacc3a6239f36e76cc8cf4f3447285bb06fc6a0 CDF + Vacuum tests This PR adds two tests testing CDF + VACUUM integration. Closes delta-io#1177 GitOrigin-RevId: f2f7b187cb3cc78c267d378eaf3d0657a56241d9 Adds miscellaneous (e.g. end-to-end workload, and CDCReader) tests for CDF. Resolves delta-io#1178 GitOrigin-RevId: 5c7da4ff9413d84e73137a80673872065de8267b Minor refactor to UpdateCommand GitOrigin-RevId: ab3fcbe0522aa194ac0781730a50112655d5c7ec Fixes delta-io#348 Support Dynamic Partition Overwrite The goal of this PR to to support dynamic partition overwrite mode on writes to delta. To enable this on a per write add `.option("partitionOverwriteMode", "dynamic")`. It can also be set per sparkSession in the SQL Config using `.config("spark.sql.sources.partitionOverwriteMode", "dynamic")`. Some limitations of this pullreq: Dynamic partition overwrite mode in combination with replaceWhere is not supported. If both are set this will result in an error. The SQL `INSERT OVERWRITE` syntax does not yet support dynamic partition overwrite. For this more changes will be needed to be made to `org.apache.spark.sql.delta.catalog.DeltaTableV2` and related classes. Fixes delta-io#348 Closes delta-io#371 Signed-off-by: Allison Portis <allison.portis@databricks.com> GitOrigin-RevId: 5b01e5b04e573dabe91ac2d71991a127617b8038 Add Checkpoint + CDF test. This PR adds a test to ensure CDC fields are not included in the checkpoint file. Resolves delta-io#1180 GitOrigin-RevId: d4a7b8bc4d1a79d30806ff18c5b507f6edcd964c
jbguerraz
pushed a commit
to jbguerraz/delta
that referenced
this pull request
Jul 6, 2022
This PR adds two tests testing CDF + VACUUM integration. Closes delta-io#1177 GitOrigin-RevId: f2f7b187cb3cc78c267d378eaf3d0657a56241d9
jbguerraz
pushed a commit
to jbguerraz/delta
that referenced
this pull request
Jul 6, 2022
This PR adds two tests testing CDF + VACUUM integration. Closes delta-io#1177 GitOrigin-RevId: f2f7b187cb3cc78c267d378eaf3d0657a56241d9
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See the project plan at #1105.
This PR adds two tests to DeltaVacuumSuite testing CDF + VACUUM integration.