-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Closed
Labels
enhancementNew feature or requestNew feature or request
Description
Is your feature request related to a problem or challenge?
FileScanConfig has statistics (FileScanConfig::statistics
) but so does file_source
datafusion/datafusion/datasource/src/file_scan_config.rs
Lines 151 to 153 in cf2b7e6
/// Estimated overall statistics of the files, taking `filters` into account. | |
/// Defaults to [`Statistics::new_unknown`]. | |
pub statistics: Statistics, |
And
datafusion/datafusion/datasource/src/file.rs
Lines 59 to 60 in cf2b7e6
/// Return projected statistics | |
fn statistics(&self) -> datafusion_common::Result<Statistics>; |
The fact there are two sets of statistics means
- there is a potential for bugs when they get out of sync such as was caused in bug: Physical plan round trip fails in some cases after datasource refactor #14679
- Planning takes that much longer
Describe the solution you'd like
It would be nice to remove the duplication so it is clear there is only a single statistics (held on the DataSource)
Describe alternatives you've considered
No response
Additional context
No response
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request