Skip to content

[SPARK-3237][SQL] Fix parquet filters with UDFs #2153

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
marmbrus wants to merge 2 commits into from
Closed

[SPARK-3237][SQL] Fix parquet filters with UDFs #2153

marmbrus wants to merge 2 commits into from

Conversation

@marmbrus
Copy link
Contributor

@marmbrus marmbrus commented Aug 27, 2014

No description provided.

@SparkQA
Copy link

SparkQA commented Aug 27, 2014

QA tests have started for PR 2153 at commit 712731a.

  • This patch merges cleanly.

@aarondav
Copy link
Contributor

aarondav commented Aug 27, 2014

LGTM after tests pass.

@SparkQA
Copy link

SparkQA commented Aug 27, 2014

QA tests have finished for PR 2153 at commit 712731a.

  • This patch passes unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • "$FWDIR"/bin/spark-submit --class $CLASS "$
    • class ExternalSorter(object):
    • "$FWDIR"/bin/spark-submit --class $CLASS "$
    • protected class AttributeEquals(val a: Attribute)

@aarondav
Copy link
Contributor

aarondav commented Aug 27, 2014

LGTM. Feel free to merge, and into branch-1.1 if you think it should go there.

@marmbrus
Copy link
Contributor Author

marmbrus commented Aug 27, 2014

Thanks for looking this over! Merged to master and 1.1.

asfgit pushed a commit that referenced this pull request Aug 27, 2014
@marmbrus
[SPARK-3237][SQL] Fix parquet filters with UDFs
ca01de1 
Author: Michael Armbrust <michael@databricks.com>

Closes #2153 from marmbrus/parquetFilters and squashes the following commits:

712731a [Michael Armbrust] Use closure serializer for sending filters.
1e83f80 [Michael Armbrust] Clean udf functions.

(cherry picked from commit e1139dd)
Signed-off-by: Michael Armbrust <michael@databricks.com>
@asfgit asfgit closed this in e1139dd Aug 27, 2014
@marmbrus marmbrus deleted the parquetFilters branch August 27, 2014 20:47
@marmbrus marmbrus mentioned this pull request Aug 28, 2014
Closed
asfgit pushed a commit that referenced this pull request Aug 28, 2014
@marmbrus @pwendell
[HOTFIX][SQL] Remove cleaning of UDFs
024178c 
It is not safe to run the closure cleaner on slaves.  #2153 introduced this which broke all UDF execution on slaves.  Will re-add cleaning of UDF closures in a follow-up PR.

Author: Michael Armbrust <michael@databricks.com>

Closes #2174 from marmbrus/fixUdfs and squashes the following commits:

55406de [Michael Armbrust] [HOTFIX] Remove cleaning of UDFs
asfgit pushed a commit that referenced this pull request Aug 28, 2014
@marmbrus @pwendell
[HOTFIX][SQL] Remove cleaning of UDFs
233c283 
It is not safe to run the closure cleaner on slaves.  #2153 introduced this which broke all UDF execution on slaves.  Will re-add cleaning of UDF closures in a follow-up PR.

Author: Michael Armbrust <michael@databricks.com>

Closes #2174 from marmbrus/fixUdfs and squashes the following commits:

55406de [Michael Armbrust] [HOTFIX] Remove cleaning of UDFs
(cherry picked from commit 024178c)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
xiliu82 pushed a commit to xiliu82/spark that referenced this pull request Sep 4, 2014
@marmbrus @conviva-zz
[SPARK-3237][SQL] Fix parquet filters with UDFs
5a4b73f 
Author: Michael Armbrust <michael@databricks.com>

Closes apache#2153 from marmbrus/parquetFilters and squashes the following commits:

712731a [Michael Armbrust] Use closure serializer for sending filters.
1e83f80 [Michael Armbrust] Clean udf functions.
xiliu82 pushed a commit to xiliu82/spark that referenced this pull request Sep 4, 2014
@marmbrus @conviva-zz
[HOTFIX][SQL] Remove cleaning of UDFs
fa9ca91 
It is not safe to run the closure cleaner on slaves.  apache#2153 introduced this which broke all UDF execution on slaves.  Will re-add cleaning of UDF closures in a follow-up PR.

Author: Michael Armbrust <michael@databricks.com>

Closes apache#2174 from marmbrus/fixUdfs and squashes the following commits:

55406de [Michael Armbrust] [HOTFIX] Remove cleaning of UDFs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@marmbrus @SparkQA @aarondav