-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-25629][TEST] Reduce ParquetFilterSuite: filter pushdown test time costs in Jenkins #22636
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Test build #96962 has finished for PR 22636 at commit
|
|
retest this please |
|
Test build #96965 has finished for PR 22636 at commit
|
|
retest this please |
|
Test build #96976 has finished for PR 22636 at commit
|
|
The time reduction is not obvious. Let us keep this unchanged? |
|
Yea, it is not obvious and only few seconds - might not be so worth. But looks improvement because it fixes the test cases to test what the previous PR targeted. Wouldn't it be better just to go ahead rather then close this since the PR is already open? |
|
@wangyum, can you reopen this? I don't think we should just close this. Fixing tests to actually test what the previous change targeted sounds definitely an improvement although it's minor. I wouldn't encourage to fix those but at least I would go ahead for this one since the fix is already proposed. |
|
Test build #97362 has finished for PR 22636 at commit
|
|
Merged to master. |
…ime costs in Jenkins ## What changes were proposed in this pull request? Only test these 4 cases is enough: https://github.com/apache/spark/blob/be2238fb502b0f49a8a1baa6da9bc3e99540b40e/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetWriteSupport.scala#L269-L279 ## How was this patch tested? Manual tests on my local machine. before: ``` - filter pushdown - decimal (13 seconds, 683 milliseconds) ``` after: ``` - filter pushdown - decimal (9 seconds, 713 milliseconds) ``` Closes apache#22636 from wangyum/SPARK-25629. Authored-by: Yuming Wang <yumwang@ebay.com> Signed-off-by: hyukjinkwon <gurwls223@apache.org>
What changes were proposed in this pull request?
Only test these 4 cases is enough:
spark/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetWriteSupport.scala
Lines 269 to 279 in be2238f
How was this patch tested?
Manual tests on my local machine.
before:
after: