[SPARK-13464][Streaming][PySpark] Fix failed streaming in pyspark in branch 1.3

[viirya]

JIRA: https://issues.apache.org/jira/browse/SPARK-13464

What changes were proposed in this pull request?

During backport a mllib feature, I found that the clearly checkouted branch-1.3 codebase would fail at the test test_reduce_by_key_and_window_with_none_invFunc in pyspark/streaming. We should fix it.

How was the this patch tested?

Unit test test_reduce_by_key_and_window_with_none_invFunc is fixed.

[viirya]

This failed test:

def test_reduce_by_key_and_window_with_none_invFunc(self):
    input = [range(1), range(2), range(3), range(4), range(5), range(6)]

    def func(dstream):
        return dstream.map(lambda x: (x, 1))\
            .reduceByKeyAndWindow(operator.add, None, 5, 1)\
            .filter(lambda kv: kv[1] > 0).count()

    expected = [[2], [4], [6], [6], [6], [6]]
    self._test_func(input, func, expected)

The input should be [0], [0, 1], [0, 1, 2], [0, 1, 2, 3], [0, 1, 2, 3, 4], [0, 1, 2, 3, 4, 5].
The key and value for each window and corresponding result after reduce func should be:

window 1: [0, 1] -> [0, 1]
window 2: [0, 1], [1, 1] -> [0, 2], [1, 1]
window 3: [0, 1], [1, 1], [2, 1] -> [0, 3], [1, 2], [2, 1]
window 4: [0, 1], [1, 1], [2, 1], [3, 1] -> [0, 4], [1, 3], [2, 2], [3, 1]
window 5: [0, 1], [1, 1], [2, 1], [3, 1], [4, 1] -> [0, 5], [1, 4], [2, 3], [3, 2], [4, 1]
window 6: [0, 1], [1, 1], [2, 1], [3, 1], [4, 1], [5, 1] -> [0, 6], [1, 5], [2, 4], [3, 3], [4, 2], [5, 1]

After filter operation and count, we actually get:

window 1: 1
window 2: 2
window 3: 3
window 4: 4
window 5: 5
window 6: 6

[viirya]

cc @mengxr

[SparkQA]

Test build #51843 has finished for PR 11339 at commit b0e48d9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

Why it passes all tests but shows test failed?

[viirya]

retest this please.

[SparkQA]

Test build #51852 has finished for PR 11339 at commit b0e48d9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

@rxin any idea about why it reports test failed even all tests are passed? Thanks.

[viirya]

From jenkins console output, it shows Build step 'Execute shell' marked build as failure.

[rxin]

It's probably because some earlier test didn't actually finish. This happens a lot with streaming leaks.

[viirya]

Is it because some streaming tests not actually passes? Not sure how to detect this.

[rxin]

cc @zsxwing

BTW why are we backporting stuff into 1.3?

[zsxwing]

retest this please

[zsxwing]

LGTM

[zsxwing]

@rxin I think it's because #9775 was merged before releasing 1.6.

[SparkQA]

Test build #51907 timed out for PR 11339 at commit b0e48d9 after a configured wait of 120m.

[viirya]

retest this please.

[SparkQA]

Test build #51915 has finished for PR 11339 at commit b0e48d9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

Hmm, what the step Execute shell actually does? It continues to pass all tests but fail at this step...

[viirya]

@zsxwing any idea about this? I think this change should not cause any failure like that. This just changes the expected result for one test.

[zsxwing]

retest this please

[SparkQA]

Test build #51979 has finished for PR 11339 at commit b0e48d9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51979/
Test FAILed.

[zsxwing]

Okey, I'm going to merge this because all Python tests passed. Thanks, @viirya

[zsxwing]

@viirya I have merged your pr. Could you close this one? Thanks!

[viirya]

@zsxwing thanks!