[Spark] Add support for sorting within partitions when Z-ordering #4006
Conversation
Signed-off-by: Malte Velin <maltevelin@proprotonmail.ch>
…value. Signed-off-by: Malte Velin <maltevelin@proprotonmail.ch>
…ording to curve. Signed-off-by: Malte Velin <maltevelin@proprotonmail.ch>
|
Hi @vkorukanti , @tdas , @scottsand-db , could you kindly review this PR when you have a moment? It introduces a configurable enhancement for Z-ordering that significantly improves read performance. I'm tagging you since you've reviewed and/or implemented PRs related to issue #1134. Your feedback would be highly appreciated! |
zachschuermann
requested review from
vkorukanti,
tdas,
scottsand-db and
rahulsmahadev
There was a problem hiding this comment.
Thanks for doing this - @zedtang do you want to also take a look?
| } | ||
| else { |
There was a problem hiding this comment.
| } | |
| else { | |
| } else { |
nit
| .internal() | ||
| .doc("If enabled, partitions are sorted on Z-order values for MDC. " + | ||
| "This co-locates records with the same Z-order values in row groups, " + | ||
| "which enables data skipping on the Parquet level.") |
There was a problem hiding this comment.
| "which enables data skipping on the Parquet level.") | |
| "which facilitates data skipping on the Parquet level.") |
Nit: We don't want to imply that row-group skipping is disabled without this conf enabled.
| @@ -74,12 +74,13 @@ trait SpaceFillingCurveClustering extends MultiDimClustering { | |||
| val conf = df.sparkSession.sessionState.conf | |||
| val numRanges = conf.getConf(DeltaSQLConf.MDC_NUM_RANGE_IDS) | |||
There was a problem hiding this comment.
Aside: we might want to increase the number of ranges when sorting within partitions when the row group size << the file size to increase the cardinality of the evaluated clustering expression
Which Delta project/connector is this regarding?
Description
Resolves #4000 by introducing a new configuration property
spark.databricks.io.skipping.mdc.sortWithinPartitionsthat clusters records in row groups, within Parquet files, based on Z-order or Hilbert curve values. This improves data skipping on the Parquet level. Benchmarks included in the issue demonstrate speedups of approximately 8× and 11× on two different datasets. Please refer to the issue for more details.How was this patch tested?
Added test cases in
MultiDimClusteringSuite.scalafor Hilbert and Z-order curves.Does this PR introduce any user-facing changes?
Yes. This PR introduces a new configuration property
spark.databricks.io.skipping.mdc.sortWithinPartitions. The property defaults tofalse, ensuring that existing users remain unaffected unless they opt-in by setting it totrue.Previous Behavior
Z-ordering did not sort data within partitions.
New Behavior
When the property is enabled,
sortWithinPartitionsis applied afterrepartitionByRangeinMultiDimClustering.scala.