Skip to content

Conversation

@liancheng
Copy link
Contributor

What changes were proposed in this pull request?

Test case initialization order under Maven and SBT are different. Maven always creates instances of all test cases and then run them all together.

This fails ObjectHashAggregateSuite because the randomized test cases there register a temporary Hive function right before creating a test case, and can be cleared while initializing other successive test cases. In SBT, this is fine since the created test case is executed immediately after creating the temporary function.

To fix this issue, we should put initialization/destruction code into beforeAll() and afterAll().

How was this patch tested?

Existing tests.

@SparkQA
Copy link

SparkQA commented Nov 8, 2016

Test build #68306 has finished for PR 15802 at commit 7b9a3cb.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@yhuai
Copy link
Contributor

yhuai commented Nov 8, 2016

Seems it does not work with sbt?

@zsxwing
Copy link
Member

zsxwing commented Nov 8, 2016

Could you add [test-maven] to the title to run the PR with maven build?

@zsxwing
Copy link
Member

zsxwing commented Nov 8, 2016

Seems it does not work with sbt?

I saw the same java.lang.OutOfMemoryError: Java heap space in my PR as well. I think it's just because these tests are flaky.

@liancheng liancheng changed the title [SPARK-18338][SQL] Fix test case initialization order under Maven builds [SPARK-18338][SQL][test-maven] Fix test case initialization order under Maven builds Nov 8, 2016
@liancheng
Copy link
Contributor Author

@cloud-fan already reported the OOM issue. I'm trying to reproduce it locally.

Added the [test-maven] tag to trigger Maven tests.

@liancheng
Copy link
Contributor Author

retest this please

@SparkQA
Copy link

SparkQA commented Nov 8, 2016

Test build #68355 has finished for PR 15802 at commit 7b9a3cb.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@liancheng
Copy link
Contributor Author

retest this please.

@liancheng
Copy link
Contributor Author

The last build failure was caused by an irrelevant flaky test.

BTW, I've reproduced the OOM issue locally by running ObjectHashAggregateSuite 200 times within a single SBT REPL session. Analyzing the heap dump.

@SparkQA
Copy link

SparkQA commented Nov 9, 2016

Test build #68365 has finished for PR 15802 at commit 7b9a3cb.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@yhuai
Copy link
Contributor

yhuai commented Nov 9, 2016

I am going to merge this to fix maven build.

@asfgit asfgit closed this in 205e6d5 Nov 9, 2016
@liancheng liancheng deleted the fix-flaky-object-hash-agg-suite branch November 9, 2016 18:21
uzadude pushed a commit to uzadude/spark that referenced this pull request Jan 27, 2017
…er Maven builds

## What changes were proposed in this pull request?

Test case initialization order under Maven and SBT are different. Maven always creates instances of all test cases and then run them all together.

This fails `ObjectHashAggregateSuite` because the randomized test cases there register a temporary Hive function right before creating a test case, and can be cleared while initializing other successive test cases. In SBT, this is fine since the created test case is executed immediately after creating the temporary function.

To fix this issue, we should put initialization/destruction code into `beforeAll()` and `afterAll()`.

## How was this patch tested?

Existing tests.

Author: Cheng Lian <lian@databricks.com>

Closes apache#15802 from liancheng/fix-flaky-object-hash-agg-suite.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants