Can I read DataColumnStatistics of the column only before reading the entire column data ?

[mirosuav]

Hi

Is there a way to read only the DataColumnStatistics before actually loading the entire column data into memory ?
Essentially I have a method that checks if the search value exists in the data column by checking column Min and Max value

public static bool ValueExistsInColumnRange<T>(this DataColumn column, T value)
        where T : IComparable<T>
    {
        if (value is null || column.Statistics.MinValue is null || column.Statistics.MaxValue is null ||
            ((T)column.Statistics.MinValue).CompareTo(value) > 0 || ((T)column.Statistics.MaxValue).CompareTo(value) < 0)
            return false;

        return true;
    }

and if value doesn't happen to exist in within the column then I can skip entire column without loading its vaules into memory. However I've noticed that
ParquetRowGroupReader.ReadColumnAsync
loads the entire column data into memory.
How to only load column statistics and optionaly load column data on demand ?

[aloneguid]

No, not now, but it should be easy to implement.

[aloneguid]

@mirosuav You should be able to access RowGroupReader.ThriftRowGroup metadata. It's a bit rough and may not suit your use case.

[mirosuav]

Thanks, @aloneguid. I already have a working solution for that, will PR it once I'm done testing.

[aloneguid]

@mirosuav just wondering, is this in any way related to GeoParquet?

[mirosuav]

@aloneguid no, I don't know GeoParquet :) We're doing our own research on comparing different storage format for big real time data.