[SPARK-16980][SQL] Load only catalog table partition metadata required to answer a query#14690

Closed

mallman wants to merge 21 commits intoapache:masterfrom

VideoAmp:spark-16980-lazy_partition_fetching

Commits on Oct 14, 2016

[SPARK-16980][SQL] Load only catalog table partition metadata required
Michael Allman
committed
Add a new catalyst optimizer rule to SQL core for pruning unnecessary
Michael Allman
committed
Include the type of file catalog in the FileSourceScanExec metadata
Michael Allman
committed
try out parquet case insensitive fallback

ericl
authored and
Michael Allman
committed
Refactor the FileSourceScanExec.metadata val to make it prettier
Michael Allman
committed
fix and add test for input files

ericl
authored and
Michael Allman
committed
rename test

ericl
authored and
Michael Allman
committed
Refactor `TableFileCatalog.listFiles` to call `listDataLeafFiles` once
Michael Allman
committed
feature flag

ericl
authored and
Michael Allman
committed
add comments

ericl
authored and
Michael Allman
committed
fix it

ericl
authored and
Michael Allman
committed
more test cases

ericl
authored and
Michael Allman
committed
also fix a bug with zero partitions selected

ericl
authored and
Michael Allman
committed
extend and fix flakiness in test

ericl
authored and
Michael Allman
committed
Enhance `ParquetMetastoreSuite` with mixed-case partition columns
Michael Allman
committed
Tidy up a little by removing some unused imports, an unused method and
Michael Allman
committed
Put partition count in `FileSourceScanExec.metadata` for partitioned
Michael Allman
committed
Fix some errors in my revision of `ParquetSourceSuite`
Michael Allman
committed
Add metrics and cost tests for partition pruning effectiveness (#5)

ericl
authored and
Michael Allman
committed
Actually register the hive catalog metrics, also revert broken tests (#6)

ericl
authored and
Michael Allman
committed
Also support mixed case field resolution for converted ORC tables (#7)

ericl
authored and
Michael Allman
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-16980][SQL] Load only catalog table partition metadata required to answer a query#14690

[SPARK-16980][SQL] Load only catalog table partition metadata required to answer a query#14690
mallman wants to merge 21 commits intoapache:masterfrom
VideoAmp:spark-16980-lazy_partition_fetching

Commits on Oct 14, 2016

[SPARK-16980][SQL] Load only catalog table partition metadata required

Add a new catalyst optimizer rule to SQL core for pruning unnecessary

Include the type of file catalog in the FileSourceScanExec metadata

try out parquet case insensitive fallback

Refactor the FileSourceScanExec.metadata val to make it prettier

fix and add test for input files

rename test

Refactor `TableFileCatalog.listFiles` to call `listDataLeafFiles` once

feature flag

add comments

fix it

more test cases

also fix a bug with zero partitions selected

extend and fix flakiness in test

Enhance `ParquetMetastoreSuite` with mixed-case partition columns

Tidy up a little by removing some unused imports, an unused method and

Put partition count in `FileSourceScanExec.metadata` for partitioned

Fix some errors in my revision of `ParquetSourceSuite`

Add metrics and cost tests for partition pruning effectiveness (#5)

Actually register the hive catalog metrics, also revert broken tests (#6)

Also support mixed case field resolution for converted ORC tables (#7)