Filter predicates should work with missing columns #1900
Description
This issue originates from SPARK-11103, which contains detailed information about how to reproduce it.
The major problem here is that, filter predicates pushed down assert that columns they touch must exist in the target physical files. But this isn't true in case of schema merging.
Actually this assertion is unnecessary, because if a column is missing in the filter schema, the column is considered to be filled by nulls, and all the filters should be able to act accordingly. For example, if we push down a = 1 but a is missing in the underlying physical file, all records in this file should be dropped since a is always null. On the other hand, if we push down a IS NULL, all records should be preserved.
Reporter: Cheng Lian / @liancheng
Assignee: Ryan Blue / @rdblue
Related issues:
- Parquet filters push-down may cause exception when schema merging is turned on (relates to)
- Parquet predicate pushdown on columns with dots return empty results (relates to)
- Cannot filter by nonexisting column in parquet file (is related to)
PRs and other links:
Note: This issue was originally created as PARQUET-389. Please see the migration documentation for further details.