You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
To Reproduce
Set the stats size to 1 (byte) and notice that statistics are still happily created
Expected behavior
The statistics size limit should be respected (and documented more carefully -- like is it the total size of all statistics? If the limit is exceeded will partial statistics have been written, etc). Looking at the java or C++ parquet writer for inspiration is likely a good idea.
Additional context
In #2022@tustvold fixed the writer to respect the "do/don't compute stats", however the "max size of computed statistics" setting is still ignored
The text was updated successfully, but these errors were encountered:
Describe the bug
WriterProperties::max_statistics_size
is ignored in the writerhttps://docs.rs/parquet/17.0.0/parquet/file/properties/struct.WriterProperties.html#method.max_statistics_size
To Reproduce
Set the stats size to
1
(byte) and notice that statistics are still happily createdExpected behavior
The statistics size limit should be respected (and documented more carefully -- like is it the total size of all statistics? If the limit is exceeded will partial statistics have been written, etc). Looking at the java or C++ parquet writer for inspiration is likely a good idea.
Additional context
In #2022 @tustvold fixed the writer to respect the "do/don't compute stats", however the "max size of computed statistics" setting is still ignored
The text was updated successfully, but these errors were encountered: