[SPARK-8564][Streaming]Add the Python API for Kinesis #6955

zsxwing · 2015-06-23T15:31:25Z

This PR adds the Python API for Kinesis, including a Python example and a simple unit test.

JoshRosen · 2015-06-23T16:24:09Z

python/run-tests

Would it be possible to move this logic into pyspark/streaming/test.py's main method instead? I'm in the middle of a Python test runner refactoring and it would greatly simplify things if the test suites themselves enforced that their prerequisites were installed rather than pushing that responsibility into the test triggering script.

Saw your refactoring work. I will update it once your PR is merged.

SparkQA · 2015-06-23T17:55:49Z

Test build #35554 has finished for PR 6955 at commit 4957c0b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class KinesisUtils(object):
- class InitialPositionInStream(object):

DoHe · 2015-07-21T14:20:51Z

python/pyspark/streaming/kinesis.py

The version is never used in the error message. This fails with:
TypeError: not all arguments converted during string formatting

SparkQA · 2015-07-23T13:34:41Z

Test build #38223 has finished for PR 6955 at commit 687446b.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class KinesisUtils(object):
- class InitialPositionInStream(object):

SparkQA · 2015-07-23T16:49:01Z

Test build #38230 has finished for PR 6955 at commit 466b425.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-07-23T17:57:28Z

Test build #38236 has finished for PR 6955 at commit 7c5cfb0.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-07-24T07:28:01Z

Test build #38312 has finished for PR 6955 at commit 6c37395.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class KinesisUtils(object):
- class InitialPositionInStream(object):

SparkQA · 2015-07-24T10:37:41Z

Test build #38341 has finished for PR 6955 at commit 96670ff.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class KinesisUtils(object):
- class InitialPositionInStream(object):

SparkQA · 2015-07-24T16:29:40Z

Test build #38353 has finished for PR 6955 at commit bcefc6e.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class KinesisUtils(object):
- class InitialPositionInStream(object):

SparkQA · 2015-07-25T03:04:39Z

Test build #38416 has finished for PR 6955 at commit b810127.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class KinesisUtils(object):
- class InitialPositionInStream(object):

SparkQA · 2015-07-25T12:24:06Z

Test build #38426 has finished for PR 6955 at commit fca416b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class KinesisUtils(object):
- class InitialPositionInStream(object):

zsxwing · 2015-07-25T15:45:49Z

retest this please

SparkQA · 2015-07-25T17:41:14Z

Test build #100 has finished for PR 6955 at commit fca416b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class KinesisUtils(object):
- class InitialPositionInStream(object):

SparkQA · 2015-07-25T17:48:44Z

Test build #38433 has finished for PR 6955 at commit fca416b.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class KinesisUtils(object):
- class InitialPositionInStream(object):

SparkQA · 2015-07-26T06:53:18Z

Test build #38449 timed out for PR 6955 at commit 5082d28 after a configured wait of 175m.

zsxwing · 2015-07-26T07:04:40Z

retest this please

SparkQA · 2015-07-26T09:25:08Z

Test build #104 has finished for PR 6955 at commit 5082d28.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class KinesisUtils(object):
- class InitialPositionInStream(object):

tdas · 2015-07-30T07:51:00Z

@zsxwing could you update this PR by merging with the master. And you probably need to add the environment variable thing to the new module.

…e folder to streaming_kinesis_asl module

SparkQA · 2015-07-30T09:54:06Z

Test build #39028 has finished for PR 6955 at commit 5082d28.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class KinesisUtils(object):
- class InitialPositionInStream(object):

tdas · 2015-07-30T10:31:24Z

Jenkins, test this please.

SparkQA · 2015-07-30T10:39:04Z

Test build #39032 has finished for PR 6955 at commit 455f7ea.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-07-30T12:46:57Z

Test build #39038 has finished for PR 6955 at commit 455f7ea.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public static final class FloatPrefixComparator extends PrefixComparator
- class KinesisUtils(object):
- class InitialPositionInStream(object):
- case class UnsafeExternalSort(

SparkQA · 2015-07-30T23:33:40Z

Test build #1240 has finished for PR 6955 at commit 455f7ea.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2015-07-31T00:21:54Z

LGTM. I will merge this when this build passes.

SparkQA · 2015-07-31T03:09:19Z

Test build #39127 timed out for PR 6955 at commit e42e471 after a configured wait of 175m.

zsxwing · 2015-07-31T03:58:01Z

retest this please

SparkQA · 2015-07-31T05:55:17Z

Test build #173 has finished for PR 6955 at commit e42e471.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-07-31T06:09:04Z

Test build #39149 has finished for PR 6955 at commit e42e471.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class KinesisUtils(object):
- class InitialPositionInStream(object):

zsxwing · 2015-07-31T06:38:12Z

retest this please

SparkQA · 2015-07-31T06:41:37Z

Test build #174 has started for PR 6955 at commit e42e471.

SparkQA · 2015-07-31T08:48:56Z

Test build #39175 has finished for PR 6955 at commit e42e471.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class KinesisUtils(object):
- class InitialPositionInStream(object):

zsxwing · 2015-07-31T09:02:33Z

retest this please

SparkQA · 2015-07-31T09:21:37Z

Test build #1255 has finished for PR 6955 at commit e42e471.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

zsxwing · 2015-07-31T09:41:32Z

retest this please

SparkQA · 2015-07-31T11:05:08Z

Test build #176 has finished for PR 6955 at commit e42e471.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class KinesisUtils(object):
- class InitialPositionInStream(object):

SparkQA · 2015-07-31T11:29:25Z

Test build #39201 has finished for PR 6955 at commit e42e471.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class KinesisUtils(object):
- class InitialPositionInStream(object):

SparkQA · 2015-07-31T11:41:01Z

Test build #177 has finished for PR 6955 at commit e42e471.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-07-31T12:04:23Z

Test build #39205 has finished for PR 6955 at commit e42e471.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class KinesisUtils(object):
- class InitialPositionInStream(object):

zsxwing · 2015-07-31T13:30:21Z

Test build #177 has finished for PR 6955 at commit e42e471.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

@tdas looks SlowSparkPullRequestBuilder doesn't have the AWS keys.

tdas · 2015-07-31T19:06:10Z

Yeah, it does not. Other tests passed, so I am merging this to master. Thanks!

Add the Python API for Kinesis

4957c0b

JoshRosen reviewed Jun 23, 2015
View reviewed changes

DoHe reviewed Jul 21, 2015
View reviewed changes

zsxwing added 3 commits July 23, 2015 02:40

Merge branch 'master' into kinesis-python

add2beb

Fix the error message and the maven output path

687446b

Fix the kinesis folder

3da2601

zsxwing added 4 commits July 23, 2015 22:19

Merge remote-tracking branch 'origin/master' into kinesis-python

e33d505

Add python tests for Kinesis

466b425

Fix the python test errors

cc9d071

RUN_KINESIS_TESTS -> ENABLE_KINESIS_TESTS

7c5cfb0

Print stack trace for debug

6c37395

zsxwing added 2 commits July 24, 2015 16:08

Merge branch 'master' into kinesis-python

756a128

Fix the compilation error after merging master

96670ff

Fix wrong comparison

fca416b

Fix the syntax error for Python 2.6

5082d28

zsxwing added 2 commits July 30, 2015 16:10

Merge remote-tracking branch 'origin/master' into kinesis-python

32e6451

Remove streaming_kinesis_asl_assembly module and simply add the sourc…

455f7ea

…e folder to streaming_kinesis_asl module

Merge branch 'master' into kinesis-python

e42e471

asfgit closed this in 3afc1de Jul 31, 2015

zsxwing deleted the kinesis-python branch August 1, 2015 01:25

[SPARK-8564][Streaming]Add the Python API for Kinesis #6955

[SPARK-8564][Streaming]Add the Python API for Kinesis #6955

Uh oh!

Conversation

zsxwing commented Jun 23, 2015

Uh oh!

JoshRosen Jun 23, 2015

Choose a reason for hiding this comment

Uh oh!

zsxwing Jun 24, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jun 23, 2015

Uh oh!

DoHe Jul 21, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 23, 2015

Uh oh!

SparkQA commented Jul 23, 2015

Uh oh!

SparkQA commented Jul 23, 2015

Uh oh!

SparkQA commented Jul 24, 2015

Uh oh!

SparkQA commented Jul 24, 2015

Uh oh!

SparkQA commented Jul 24, 2015

Uh oh!

SparkQA commented Jul 25, 2015

Uh oh!

SparkQA commented Jul 25, 2015

Uh oh!

zsxwing commented Jul 25, 2015

Uh oh!

SparkQA commented Jul 25, 2015

Uh oh!

SparkQA commented Jul 25, 2015

Uh oh!

SparkQA commented Jul 26, 2015

Uh oh!

zsxwing commented Jul 26, 2015

Uh oh!

SparkQA commented Jul 26, 2015

Uh oh!

tdas commented Jul 30, 2015

Uh oh!

SparkQA commented Jul 30, 2015

Uh oh!

tdas commented Jul 30, 2015

Uh oh!

SparkQA commented Jul 30, 2015

Uh oh!

SparkQA commented Jul 30, 2015

Uh oh!

SparkQA commented Jul 30, 2015

Uh oh!

tdas commented Jul 31, 2015

Uh oh!

SparkQA commented Jul 31, 2015

Uh oh!

zsxwing commented Jul 31, 2015

Uh oh!

SparkQA commented Jul 31, 2015

Uh oh!

SparkQA commented Jul 31, 2015

Uh oh!

zsxwing commented Jul 31, 2015

Uh oh!

SparkQA commented Jul 31, 2015

Uh oh!

SparkQA commented Jul 31, 2015

Uh oh!

zsxwing commented Jul 31, 2015

Uh oh!

SparkQA commented Jul 31, 2015

Uh oh!

zsxwing commented Jul 31, 2015

Uh oh!

SparkQA commented Jul 31, 2015