Commits on Aug 5, 2024

1. [Spark] Fix the semantic of shouldRewriteToBeIcebergCompatible in R… …

   …EORG UPGRADE UNIFORM (delta-io#3412)
   
   ## Description
   currently we utilize the helper function
   `shouldRewriteToBeIcebergCompatible` to filter the portion of parquet
   files that need to be rewritten when running `REORG UPGRADE UNIFORM`
   based on the tags in the `AddFile`.
   
   however, the `DeltaUpgradeUniformOperation.icebergCompatVersion` is
   accidentally shadowed, which will make
   `shouldRewriteToBeIcebergCompatible` always return `false` if the
   `AddFile.tags` is not `null` - this is not the expected semantic of this
   function.
   
   this PR introduces the fix for this problem and add unit tests to ensure
   the correctness.
   
   ## How was this patch tested?
   through unit tests in `UniFormE2ESuite.scala`.
   
   ## Does this PR introduce _any_ user-facing changes?
   no.

   

   xzhseh committed Aug 5, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for c530533
   • Browse repository at this point

   Copy the full SHA

   c530533 View commit details

   Browse the repository at this point in the history