Skip to content

Conversation

@ueshin
Copy link
Member

@ueshin ueshin commented Jun 16, 2016

What changes were proposed in this pull request?

This pr adds PushPredicateThroughObjectConsumer rule to push-down predicates through ObjectConsumer.
And as an example, I implemented push-down typed filter through SerializeFromObject.

The optimized plan for the following DataFrame:

Seq(("a", 1), ("b", 2), ("c", 3)).toDS().map(v => (v._1, v._2 + 1)).filter(_._1 == "b")

was before patch:

Filter <function1>.apply
+- SerializeFromObject [staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, assertnotnull(input[0, scala.Tuple2, true], top level non-flat input object)._1, true) AS _1#449, assertnotnull(input[0, scala.Tuple2, true], top level non-flat input object)._2 AS _2#450]
   +- MapElements <function1>, obj#448: scala.Tuple2
      +- DeserializeToObject newInstance(class scala.Tuple2), obj#447: scala.Tuple2
         +- LocalRelation [_1#442, _2#443]

becomes after patch:

SerializeFromObject [staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, assertnotnull(input[0, scala.Tuple2, true], top level non-flat input object)._1, true) AS _1#449, assertnotnull(input[0, scala.Tuple2, true], top level non-flat input object)._2 AS _2#450]
+- Filter <function1>.apply
   +- MapElements <function1>, obj#448: scala.Tuple2
      +- DeserializeToObject newInstance(class scala.Tuple2), obj#447: scala.Tuple2
         +- LocalRelation [_1#442, _2#443]

How was this patch tested?

Added tests to check if push-down typed filter through SerializeFromObject correctly works.

@SparkQA
Copy link

SparkQA commented Jun 16, 2016

Test build #60631 has finished for PR 13702 at commit 75805a4.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@ueshin
Copy link
Member Author

ueshin commented Jun 21, 2016

@marmbrus @cloud-fan
Could you review this pr please?

@ueshin
Copy link
Member Author

ueshin commented Jun 24, 2016

I'm closing this in favor of #13846.

@ueshin ueshin closed this Jun 24, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants