[SPARK-23539][SS][FOLLOWUP][TESTS] Add UT to ensure existing query doesn't break with default conf of includeHeaders #25792
Conversation
…reak with default conf of includeHeaders
|
This patch is to address my own comment which PR was merged without addressing this. |
HeartSaVioR
changed the title
|
Test build #110604 has finished for PR 25792 at commit
|
|
Actually the other intention was let test fail when "includeHeaders" is set to "true", as we suspected from #22282 (comment), but turns out it doesn't break even the option is set to true. It's because of the mechanism how Luckily it doesn't let query crash, but it would be ideal to do type check and avoid the possibility of crashing in runtime - SPARK-27237 (#24173) has been proposed to address this. |
|
cc. @zsxwing @srowen as reviewers with commit privileges of #24173 , and @dongjinleekr as author of #24173 |
| intercept[IllegalArgumentException] { test(minPartitions = "-1", 1, true) } | ||
| } | ||
|
|
||
| test("default config of includeHeader doesn't break existing query from Spark 2.4") { |
|
Test build #4868 has finished for PR 25792 at commit
|
|
FYI, CI build just got passed. |
|
Merged to master |
|
Thanks all for reviewing and merging! |
6 participants
What changes were proposed in this pull request?
This patch adds new UT to ensure existing query (before Spark 3.0.0) with checkpoint doesn't break with default configuration of "includeHeaders" being introduced via SPARK-23539.
This patch also modifies existing test which checks type of columns to also check headers column as well.
Why are the changes needed?
The patch adds missing tests which guarantees backward compatibility of the change of SPARK-23539.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
UT passed.