Add new stats pruning helpers to allow combining partition values in file level stats #16139

adriangb · 2025-05-21T15:24:51Z

A step towards #16014

adriangb · 2025-05-21T17:11:53Z

@xudong963 any chance you can review this since you've already approved the same code (with less tests!) in the original PR?

kosiew

Dropped some suggestion and comments.

kosiew · 2025-05-23T03:54:55Z

datafusion/common/src/pruning.rs

+    ) -> Self {
+        let num_containers = partition_values.len();
+        let partition_schema = Arc::new(Schema::new(partition_fields));
+        let mut partition_valeus_by_column =


Suggested change

let mut partition_valeus_by_column =

let mut partition_values_by_column =

kosiew · 2025-05-23T04:05:21Z

datafusion/common/src/pruning.rs

+    ///   The outer vector represents the containers while the inner
+    ///   vector represents the partition values for each column.


constructor accepts partition_values as a Vec, documented as “outer vector represents the containers while the inner vector represents the partition values for each column.” In code however, each inner Vec is treated as the values for one container, then transpose that into column-major storage.

The phrasing “inner vector represents the partition values for each column” can be read as “one column’s values across containers.”

kosiew · 2025-05-23T04:08:56Z

datafusion/common/src/pruning.rs

+    fn min_values(&self, column: &Column) -> Option<ArrayRef> {
+        let index = self.schema.index_of(column.name()).ok()?;
+        if self.statistics.iter().any(|s| {
+            s.column_statistics
+                .get(index)
+                .is_some_and(|stat| stat.min_value.is_exact().unwrap_or(false))
+        }) {
+            match ScalarValue::iter_to_array(self.statistics.iter().map(|s| {
+                s.column_statistics
+                    .get(index)
+                    .and_then(|stat| {
+                        if let Precision::Exact(min) = &stat.min_value {
+                            Some(min.clone())
+                        } else {
+                            None
+                        }
+                    })
+                    .unwrap_or(ScalarValue::Null)
+            })) {
+                Ok(array) => Some(array),
+                Err(_) => {
+                    log::warn!(
+                        "Failed to convert min values to array for column {}",
+                        column.name()
+                    );
+                    None
+                }
+            }
+        } else {
+            None
+        }
+    }


Both PrunableStatistics::min_values and max_values walk the same steps:

Find the column index in the schema.

Check whether any Statistics entry has an “exact” value for that column.

Iterate over all Statistics, pulling out the exact values or substituting ScalarValue::Null.

Call ScalarValue::iter_to_array(...) and log or return None on error.

By lifting steps (2)–(4) into a helper, we:

Eliminate duplicate code in each method

Centralize error handling and logging

Make future changes (e.g. using a different logging framework) in one place

kosiew · 2025-05-23T04:16:41Z

datafusion/common/src/pruning.rs

+        let mut contained = Vec::with_capacity(self.partition_values.len());
+        for partition_value in partition_values {
+            let contained_value = if values.contains(partition_value) {
+                Some(true)
+            } else {
+                Some(false)
+            };
+            contained.push(contained_value);
+        }
+        let array = BooleanArray::from(contained);


Instead of explicit loops; would simplifying to .map(...) chains followed by collect() be better?

let array = BooleanArray::from( partition_values .iter() .map(|pv| Some(values.contains(pv))) .collect::<Vec<_>>() );

Benefits:

Eliminates manual push logic

More concise: transforms each pv into a boolean directly

Clearly shows “map input → output” intent

adriangb · 2025-05-23T13:57:33Z

Thank you @kosiew that was great feedback 😄

alamb

Thank you @adriangb and @kosiew -- this looks like great work to me

datafusion/common/src/pruning.rs

alamb · 2025-05-28T15:59:59Z

datafusion/common/src/pruning.rs

+    fn min_values(&self, column: &Column) -> Option<ArrayRef> {
+        let index = self.partition_schema.index_of(column.name()).ok()?;
+        let partition_values = self.partition_values.get(index)?;
+        match ScalarValue::iter_to_array(partition_values.iter().cloned()) {


It is always sad to me that this API requires a clone of ScalarValue 😢 (nothing you did in this PR)

datafusion/common/src/pruning.rs

xudong963 · 2025-05-28T16:20:33Z

Sorry for late, I'll check tomorrow (feel free to directly invite me to review by the button, then I'll notice more)

adriangb · 2025-05-28T16:32:03Z

Sorry for late, I'll check tomorrow (feel free to directly invite me to review by the button, then I'll notice more)

I'm not able to request reviews. I think only commiters can do that and I'm not a commiter (yet).

…ngStatistics for partition + file level stats pruning

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

This reverts commit 8b6f1a2.

alamb · 2025-05-28T17:24:10Z

I'm not able to request reviews. I think only commiters can do that and I'm not a commiter (yet).

I think you will need to do the gitbox thing with your apache account (when it is activated) and then you will be an official committer

xudong963 · 2025-05-29T12:57:21Z

datafusion/common/src/pruning.rs

+}
+
+/// Prune a set of containers represented by their statistics.
+/// Each [`Statistics`] represents a container (e.g. a file or a partition of files).


What does a partition of files mean? A FileGroup or just a collection of some files?

It's any collection of files. Basically it's up to the caller to define what the container is: it could be a single file or a FileGroup. I do think it'd be helpful to link to possible sources of statistics here (e.g. link to [FileGroup]).

I will make a follow on PR

Follow on PR: #16213

xudong963 · 2025-05-29T13:17:39Z

datafusion/common/src/pruning.rs

+/// If multiple statistics have information for the same column,
+/// the first one is returned without any regard for completeness or accuracy.
+/// That is: if the first statistics has information for a column, even if it is incomplete,
+/// that is returned even if a later statistics has more complete information.


I'm curious about why not prune based on all the statistics that have information, one by one

I guess we could do that, but I'm not really sure where we'd encounter that.
The immediate use case here is having partition values for partition columns and file level statistics for the rest, so there is no overlap. But I had to choose some behavior for this implementation so I went with "first". I guess "both via AND" is reasonable, it just seems a bit harder to explain / get right. But I can give it a shot.

Yeah, we can also just add a todo and maintain the current status in the PR

alamb · 2025-05-30T10:42:32Z

Thanks @adriangb and @xudong963 -- this looks good to me 🚀

github-actions bot added the common Related to common crate label May 21, 2025

kosiew reviewed May 23, 2025

View reviewed changes

adriangb force-pushed the add-pruning-structs branch from fe0b8f1 to 8b7089f Compare May 23, 2025 13:57

adriangb force-pushed the add-pruning-structs branch from 8b7089f to 27cdc22 Compare May 26, 2025 16:35

alamb approved these changes May 28, 2025

View reviewed changes

xudong963 self-requested a review May 28, 2025 16:20

adriangb and others added 6 commits May 28, 2025 12:16

wip

407902e

Add PrunableStatistics, PartitionPruningStatistics and CompositePruni…

7fc524a

…ngStatistics for partition + file level stats pruning

pr feedback

12092ae

clippy

350cfd0

Update datafusion/common/src/pruning.rs

8b6f1a2

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Revert "Update datafusion/common/src/pruning.rs"

90304b6

This reverts commit 8b6f1a2.

use ScalarValue::from

5776221

adriangb force-pushed the add-pruning-structs branch from 5094e27 to 5776221 Compare May 28, 2025 17:24

adriangb added 3 commits May 28, 2025 13:07

broken imple using eq kernel

23e3425

fix arrays

0def286

clippy, remove collect

ea12495

xudong963 reviewed May 29, 2025

View reviewed changes

xudong963 approved these changes May 30, 2025

View reviewed changes

alamb merged commit c6e5c91 into apache:main May 30, 2025
27 checks passed

alamb mentioned this pull request May 30, 2025

Minor: update documentation for PrunableStatistics #16213

Merged

	let mut partition_valeus_by_column =
	let mut partition_values_by_column =

		/// The outer vector represents the containers while the inner
		/// vector represents the partition values for each column.

Add new stats pruning helpers to allow combining partition values in file level stats #16139

Add new stats pruning helpers to allow combining partition values in file level stats #16139

Uh oh!

Conversation

adriangb commented May 21, 2025

Uh oh!

adriangb commented May 21, 2025

Uh oh!

kosiew left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adriangb commented May 23, 2025

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

xudong963 commented May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adriangb commented May 28, 2025

Uh oh!

alamb commented May 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented May 30, 2025

Uh oh!

Uh oh!

Uh oh!

xudong963 commented May 28, 2025 •

edited

Loading