Optimize the parquet RecordReader implementation when: A. filterpredicate is pushed down , B. filterpredicate is pushed down on a flat schema

[asfimport]

The RecordReader implementation currently will read all the columns before applying the filter predicate and deciding whether to keep the row or discard it.
We can have a RecordReader which will only assemble the columns on which filters are applied (which are usually a few), then apply the filter and decide whether to keep the row or not , and then goes on to assemble the remaining columns or skip the remaining columns accordingly.

Also for applications like spark sql , the schema usually applied is a flat one with no repeating or nested columns. In such cases, its better to have a light-weight, faster RecordReader.

The performance improvement by this change is seen to be significant , and is better in case smaller number of rows are returned by filtering (which is usually the case) and there are many number of columns

Reporter: Yash Datta / @saucam

_{Note: This issue was originally created as PARQUET-128. Please see the migration documentation for further details.}

[asfimport]

Yash Datta / @saucam:
For a 47 million row parquet table in spark-sql with schema with no nested/repeating columns ; applying a simple filter :

select * from d_tup_parq_yash where id = 10; (which returns a single record)
or
select * from d_tup_parq_yash where id < 500; (which returns 500 records)

time taken before the patch : 4.4 seconds
time taken after the patch : 2.6 seconds

[asfimport]

Yash Datta / @saucam:
created a pull request : https://github.com/apache/incubator-parquet-mr/pull/82
*This has been closed in favor of a new PR mentioned below

[asfimport]

Yash Datta / @saucam:
This follows the approach mentioned in PARQUET-37 of restructuring the assembly algorithm to first read columns involved in the filter predicate, and only if it passes are the other values read.

[asfimport]

Yash Datta / @saucam:
Created PR

#288

after rebasing to the new code base