[HOTFIX] [CORE] fix flaky BasicSchedulerIntegrationTest #13688
Conversation
| case _ => | ||
| // we can't check for the output for the two intermediate stages, unfortunately, | ||
| // b/c the stage numbering is non-deterministic, so stage number alone doesn't tell | ||
| // us what to check |
There was a problem hiding this comment.
Does it make sense to do an OR here -- check that one of the intermediate outputs is available? (fine if you want to do this later and merge the fix for now)
There was a problem hiding this comment.
I thought about this, but decided it was kind of silly since you could be checking for something irrelevant, and its not a trivial change (you'd need to catch an exception from doing the wrong check). Anyway just figured I'd leave it out for the hotfix.
|
LGTM Would be great if we could (later) update these tests so they still check the same things, but in a way that doesn't rely on the stage ID being deterministic. |
|
As a HOTIFX, LGTM; but I agree that there is room for a follow-up. |
|
core tests passed so I merged to master (not 2.0). Will keep an eye on it as well. |
|
Test build #60593 has finished for PR 13688 at commit
|
|
also opened https://issues.apache.org/jira/browse/SPARK-15976 for related discussion. |
4 participants
What changes were proposed in this pull request?
SPARK-15927 exacerbated a race in BasicSchedulerIntegrationTest, so it went from very unlikely to fairly frequent. The issue is that stage numbering is not completely deterministic, but these tests treated it like it was. So turn off the tests.
How was this patch tested?
on my laptop the test failed abotu 10% of the time before this change, and didn't fail in 500 runs after the change.