Skip to content

Conversation

@eatoncys
Copy link
Contributor

@eatoncys eatoncys commented Jan 11, 2019

What changes were proposed in this pull request?

BroardCast hint can not work with PruneFileSourcePartitions, for example, when the filter condition p is a partition field, table b in SQL below cannot be broadcast.

sql("select /*+ broadcastjoin(b) */ * from (select a from empty_test where p=1) a " + "join (select a,b from par_1 where p=1) b on a.a=b.a").explain

The reason is that PruneFileSourcePartitions deleted ResolvedHint when matching PhysicalOperations, the plan after PruneFileSourcePartitions before this PR is:

image

The original reason for this problem is that when PhysicalOperation matches ResolvedHint, it returns directly to its child nodes and skips over itself.
case h: ResolvedHint => collectProjectsAndFilters(h.child)

This PR adds processing to ResolvedHint , when matched to it in PruneFileSourcePartitions, skip it directly and parse its child nodes.

How was this patch tested?

Unit test of PruneFileSourcePartitionsSuite

@SparkQA
Copy link

SparkQA commented Jan 11, 2019

Test build #101068 has finished for PR 23517 at commit f7bca2f.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@eatoncys
Copy link
Contributor Author

retest this please

@SparkQA
Copy link

SparkQA commented Jan 11, 2019

Test build #101077 has finished for PR 23517 at commit 78e19b5.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@eatoncys
Copy link
Contributor Author

It has been resolved by another pr #23507

@eatoncys eatoncys closed this Jan 11, 2019
@SparkQA
Copy link

SparkQA commented Jan 11, 2019

Test build #101071 has finished for PR 23517 at commit f7bca2f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants