[SPARK-22591][SQL] GenerateOrdering shouldn't change CodegenContext.INPUT_ROW #19800
Conversation
|
cc @cloud-fan @kiszk |
| } | ||
| } | ||
|
|
||
| test("SPARK-22591: GenerateOrdering shouldn't change ctx.INPUT_ROW") { |
There was a problem hiding this comment.
I'm working to support wholestage codegen when generating expression codes safe from 64k limit. When there is not INPUT_ROW in context but we wrongly set a INPUT_ROW value, a non-existing InternalRow i will be added into function parameters.
There was a problem hiding this comment.
As we use INPUT_ROW a lot in codegen framework, it is risky to change its value without restoring it.
| def genComparisons(ctx: CodegenContext, ordering: Seq[SortOrder]): String = { | ||
| val oldInputRow = ctx.INPUT_ROW | ||
| val comparisons = ordering.map { order => | ||
| val oldCurrentVars = ctx.currentVars |
There was a problem hiding this comment.
seems we can also move this our of the loop.
There was a problem hiding this comment.
how about
val oldInputRow = ...
val oldCurrentVars = ...
val inputRow = "i"
ctx.INPUT_ROW = inputRow
ctx.currentVars = null
val comparisons = ...
...
ctx.INPUT_ROW = oldInputRow
ctx.currentVars = oldCurrentVars
...
|
Test build #84128 has finished for PR 19800 at commit
|
|
This test |
|
Test build #84131 has finished for PR 19800 at commit
|
| // Restore original currentVars and INPUT_ROW. | ||
| ctx.currentVars = oldCurrentVars | ||
| ctx.INPUT_ROW = oldInputRow | ||
| finalCode |
There was a problem hiding this comment.
ctx.currentVars = oldCurrentVars
ctx.INPUT_ROW = oldInputRow
s"""
|InternalRow $inputRow = null;
|$code
""".stripMargin
|
I created https://issues.apache.org/jira/browse/SPARK-22595 to track the flaky test |
|
Test build #84144 has finished for PR 19800 at commit
|
|
thanks, merging to master/2.2! |
3 participants
What changes were proposed in this pull request?
When I played with codegen in developing another PR, I found the value of
CodegenContext.INPUT_ROWis not reliable. Under wholestage codegen, it is assigned to null first and then suddenly changed toi.The reason is
GenerateOrderingchangesCodegenContext.INPUT_ROWbut doesn't restore it back.How was this patch tested?
Added test.