Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create a build configuration for Spark 3 preview release(s) #2237

Closed
heuermh opened this issue Dec 20, 2019 · 2 comments
Closed

Create a build configuration for Spark 3 preview release(s) #2237

heuermh opened this issue Dec 20, 2019 · 2 comments
Milestone
0.32.0

Comments

@heuermh
Copy link
Member

heuermh commented Dec 20, 2019

No description provided.

@heuermh
Copy link
Member Author

heuermh commented Apr 29, 2020

See

[API][3.0][SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL] Nested Column Predicate Pushdown for Parquet (+852, -480)>

This PR extends the data source framework to support filter pushdown with nested fields, and implement it in the Parquet data source.

spark.sql.optimizer.nestedPredicatePushdown.enabled [Default: true]

When true, Spark tries to push down predicates for nested columns and or names containing dots to data sources. Currently, Parquet implements both optimizations while ORC only supports predicates for names containing dots. The other data sources don't support this feature yet.

@heuermh heuermh added this to the 0.32.0 milestone Jun 26, 2020
@heuermh
Copy link
Member Author

heuermh commented Jun 26, 2020

Fixed by #2255

@heuermh heuermh closed this as completed Jun 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
0.32.0
Development

No branches or pull requests
1 participant
@heuermh