ARROW-8733: [C++][Dataset][Python] Expose RowGroupInfo statistics values

[bkietz]

stats = parquet_fragment.row_groups[0].statistics
assert stats == {
  'normal_column': {'min': 1, 'max': 2},
  'all_null_column': {'min': None, 'max': None},
  'column_without_stats': None,
}

[github-actions]

https://issues.apache.org/jira/browse/ARROW-8733

[jorisvandenbossche]

Thanks, looks perfect!

The shape is fine I think, a dict of {col : {min: val, max: val}} seems the most logical / general structure to store it. And in the end, dask will need to massage it into some structure they need anyway (combining it for multiple fragments) cc @rjzamora

[jorisvandenbossche]

Do we want to expose the statistics_expression as well? (not fully sure if it would have a use case, so maybe we should only do that if we have one)

[bkietz]

If that's desired it can wait for a follow up

[fsaintjacques]

LGTM, clean solution.

[rjzamora]

Thanks for the great work here @bkietz !

This is wonderful - Dask uses the min/max statistics to calculate divisions, so this functionality is definitely necessary.


A note on other (less-critical, but useful) statistics:
Dask also uses the "total_byte_size" statistics (for the full row-group, not each column) to aggregate partitions before reading in any data. There is also a plan to use the "num-rows” statistics when the user executes len(ddf) (to avoid loading any data). How difficult would it be to add/expose these additional row-group statistics? Again, this is much less of a “blocker” for initial integration with Dask, but are likely things we will want to add in eventually. cc @jorisvandenbossche

[jorisvandenbossche]

@rjzamora num_rows is already available on the RowGroupInfo object (

arrow/python/pyarrow/_dataset.pyx

Lines 845 to 847 in cd3ed60

	@property
	def num_rows(self):
	return self.info.num_rows()

)

For the total_byte_size, can you open a JIRA for this? (it should be similar as num_rows to get / cache from the parquet row group, I think)

[jorisvandenbossche]

@rjzamora I opened https://issues.apache.org/jira/browse/ARROW-9346 to track the total_byte_size suggestion