Clarify documentation about gathering statistics for parquet files #16157
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alamb
changed the title
github-actions
bot
added
documentation
davisp
reviewed
May 25, 2025
Member
davisp
left a comment
davisp
left a comment
There was a problem hiding this comment.
Apologies for reviewing from mobile and not linking directly, but in ddl.md I think that should be collect_statistics, not show_statistics.
xudong963
reviewed
May 26, 2025
docs/source/user-guide/sql/ddl.md
Outdated
| : By default, when a table is created, DataFusion will _NOT_ read the files | ||
| to gather statistics, which can be expensive but can accelerate subsequent | ||
| queries substantially. If you want to gather statistics | ||
| when creating a table, set the `datafusion.explain.show_statistics` |
Member
There was a problem hiding this comment.
datafusion.explain.collect_statistics?
comphead
reviewed
May 27, 2025
comphead
reviewed
May 27, 2025
comphead
approved these changes
May 27, 2025
Contributor
comphead
left a comment
comphead
left a comment
There was a problem hiding this comment.
lgtm thanks @alamb
agree with @xudong963 that you probably mean another param to set for collecting stats?
in this PR
/// When set to true, the explain statement will print operator statistics
/// for physical plans
pub show_statistics: bool, default = false
was used but I dont see EXPLAIN stmts
Co-authored-by: Oleks V <comphead@users.noreply.github.com>
alamb
commented
May 27, 2025
| LOCATION '/mnt/nyctaxi/tripdata.parquet'; | ||
| ``` | ||
|
|
||
| :::{note} |
Contributor
Author
There was a problem hiding this comment.
Here is an example of what this looks like rendered
Member
There was a problem hiding this comment.
Here is an example of what this looks like rendered
TIL
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
SessionContext::register_parquetandCREATE EXTERNAL TABLE#15908 from @davispRationale for this change
As noted by @davisp it was not clear that statistics are not collected by default for ListingTables which has a potentially substantial negative impact on performance. Let's at least document this
What changes are included in this PR?
Document when statistics are (not) collected and add notes about how to enable them
Are these changes tested?
Yes by CI
Are there any user-facing changes?
Docs only