does arrow support parquet column index feathure?

[FANNG1]

parquet has columnIndex to support page skiping (https://github.com/apache/parquet-format/blob/master/PageIndex.md), does arrow support it , and if not, any plan to support it?

[westonpace]

Regarding the C++ implementation (and by extension the python, R, and Ruby extensions): parquet-C++, the parquet library that is part of (and used by) arrow-c++, does have some support for serializing and deserializing these structures.

However, Arrow's readers and writers for parquet do not (to the best of my knowledge) support using these indices for filter pushdown and do not have support for writing indices.

Arrow is an open source project and so "any plan to support it" usually boils down to whether there is someone motivated enough with enough time to tackle the feature. It is something I think would be a great addition.

Adding the feature to the C++ implementation is tracked in PARQUET-1404 and ARROW-10158 There was an attempt to implement this referenced by those JIRA tickets but, unfortunately, it appears that work may have been abandoned. There is a related mailing list discussion here.

Adding the feature to the Rust implementation is tracked here apache/datafusion#847

[FANNG1]

we're doing technical investigation about parquet and surprised to see arrow had support it，thanks for your detailed and useful reply。