[SPARK-4502][SQL] Parquet nested column pruning - foundation (2nd attempt)#21889

Closed

ajacques wants to merge 19 commits intoapache:masterfrom

ajacques:spark-4502-parquet_column_pruning-foundation

Commits on Aug 4, 2018

[SPARK-4502][SQL] Parquet nested column pruning

Michael Allman
authored and
mallman
committed
Refactor SelectedFieldSuite to make its tests simpler and more
mallman
committed
Remove test "select function over nested data" of unknown origin and
mallman
committed
Improve readability of ParquetSchemaPruning and
mallman
committed
Don't handle non-data-field partition column names specially when
mallman
committed
Add test coverage for ParquetSchemaPruning for partitioned tables whose
mallman
committed
Remove the ColumnarFileFormat type to put it in another PR
mallman
committed
Add test coverage for the enhancements to "is not null" constraint
mallman
committed
Revert changes to QueryPlanConstraints.scala and basicPhysicalOperators.scala. Remove related unit tests. This change removes support for schema pruning on filters involving attributes of complex type
mallman
committed
Revert a whitespace change in DataSourceScanExec.scala
mallman
committed
Remove modifications to ParquetFileFormat.scala and
mallman
committed
PR review: simplify some syntax and add a code doc
mallman
committed
When creating a pruned schema by merging an array of root structs, sort
mallman
committed
Re-enable ignored test in ParquetSchemaPruningSuite.scala that is
mallman
committed
Enable schema pruning by default
mallman
committed

Commits on Aug 5, 2018

Commits on Aug 8, 2018

Refactor code slightly based on code review comments
ajacques
committed

Commits on Aug 15, 2018

Clean-up code based on review comments
ajacques
committed