Skip to content

[SPARK-6514][SPARK-5960][SPARK-6656][SPARK-7679][STREAMING][KINESIS] Updates to the Kinesis API #6147

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
tdas wants to merge 2 commits into from
Closed

[SPARK-6514][SPARK-5960][SPARK-6656][SPARK-7679][STREAMING][KINESIS] Updates to the Kinesis API #6147

tdas wants to merge 2 commits into from

Conversation

@tdas
Copy link
Contributor

@tdas tdas commented May 14, 2015

SPARK-6514 - Use correct region
SPARK-5960 - Allow AWS Credentials to be directly passed
SPARK-6656 - Specify kinesis application name explicitly
SPARK-7679 - Upgrade to latest KCL and AWS SDK.

@AmplabJenkins
Copy link

AmplabJenkins commented May 14, 2015

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented May 14, 2015

Merged build started.

@SparkQA
Copy link

SparkQA commented May 14, 2015

Test build #32711 has started for PR 6147 at commit 373b201.

@tdas
Copy link
Contributor Author

tdas commented May 14, 2015

@cfregly

@SparkQA
Copy link

SparkQA commented May 14, 2015

Test build #32711 has finished for PR 6147 at commit 373b201.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented May 14, 2015

Merged build finished. Test FAILed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 14, 2015

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32711/
Test FAILed.

@tdas tdas changed the title [SPARK-6514][SPARK-5960][SPARK-6656][STREAMING][KINESIS] Updates to the Kinesis API [SPARK-6514][SPARK-5960][SPARK-6656][SPARK-7679][STREAMING][KINESIS] Updates to the Kinesis API May 15, 2015
@AmplabJenkins
Copy link

AmplabJenkins commented May 15, 2015

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented May 15, 2015

Merged build started.

@SparkQA
Copy link

SparkQA commented May 15, 2015

Test build #32860 has started for PR 6147 at commit f23ea77.

@SparkQA
Copy link

SparkQA commented May 16, 2015

Test build #32860 timed out for PR 6147 at commit f23ea77 after a configured wait of 150m.

@AmplabJenkins
Copy link

AmplabJenkins commented May 16, 2015

Merged build finished. Test FAILed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 16, 2015

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32860/
Test FAILed.

@tdas
Copy link
Contributor Author

tdas commented May 16, 2015

test this please.

@AmplabJenkins
Copy link

AmplabJenkins commented May 16, 2015

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented May 16, 2015

Merged build started.

@SparkQA
Copy link

SparkQA commented May 16, 2015

Test build #32876 has started for PR 6147 at commit f23ea77.

@SparkQA
Copy link

SparkQA commented May 16, 2015

Test build #32876 has finished for PR 6147 at commit f23ea77.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented May 16, 2015

Merged build finished. Test FAILed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 16, 2015

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32876/
Test FAILed.

@tdas
Copy link
Contributor Author

tdas commented May 16, 2015

test this please.

@AmplabJenkins
Copy link

AmplabJenkins commented May 16, 2015

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented May 16, 2015

Merged build started.

@SparkQA
Copy link

SparkQA commented May 16, 2015

Test build #32890 has started for PR 6147 at commit f23ea77.

@SparkQA
Copy link

SparkQA commented May 16, 2015

Test build #32890 timed out for PR 6147 at commit f23ea77 after a configured wait of 150m.

@AmplabJenkins
Copy link

AmplabJenkins commented May 16, 2015

Merged build finished. Test FAILed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 16, 2015

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32890/
Test FAILed.

@tdas
Copy link
Contributor Author

tdas commented May 17, 2015

test this please.

@AmplabJenkins
Copy link

AmplabJenkins commented May 17, 2015

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented May 17, 2015

Merged build started.

@SparkQA
Copy link

SparkQA commented May 17, 2015

Test build #32922 has started for PR 6147 at commit f23ea77.

@SparkQA
Copy link

SparkQA commented May 17, 2015

Test build #32922 has finished for PR 6147 at commit f23ea77.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • case class SerializableAWSCredentials(accessKeyId: String, secretKey: String)

@AmplabJenkins
Copy link

AmplabJenkins commented May 17, 2015

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 17, 2015

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32922/
Test PASSed.

@cfregly
Copy link
Contributor

cfregly commented May 17, 2015

@tdas: Looks like you forgot to bring over the docs/streaming-kinesis-integration.md and 2 examples (JavaKinesisWordCountASL and KinesisWordCountASL) changes from my original PR: https://github.com/apache/spark/pull/5882/files

how do you suggest we proceed? i believe the samples - if left unchanged - would use the old deprecated/experimental API.

@tdas
Copy link
Contributor Author

tdas commented May 17, 2015

Since this has passed, let me merge this. I will pull over the examples in another PR.

@asfgit asfgit closed this in ca4257a May 17, 2015
asfgit pushed a commit that referenced this pull request May 17, 2015
@tdas
[SPARK-6514] [SPARK-5960] [SPARK-6656] [SPARK-7679] [STREAMING] [KINE…
e0632ff 
…SIS] Updates to the Kinesis API

SPARK-6514 - Use correct region
SPARK-5960 - Allow AWS Credentials to be directly passed
SPARK-6656 - Specify kinesis application name explicitly
SPARK-7679 - Upgrade to latest KCL and AWS SDK.

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes #6147 from tdas/kinesis-api-update and squashes the following commits:

f23ea77 [Tathagata Das] Updated versions and updated APIs
373b201 [Tathagata Das] Updated Kinesis API

(cherry picked from commit ca4257a)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
cfregly
cfregly reviewed May 21, 2015
View reviewed changes
extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala
Copy link
Contributor

@cfregly cfregly May 21, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@param initialPositionInStream seems to be missing

also, it appears to have moved below checkpointInterval which is a bit confusing. i hit this while testing.

the @param docs may need to be re-ordered, as well

jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
@tdas @jeanlyn
[SPARK-6514] [SPARK-5960] [SPARK-6656] [SPARK-7679] [STREAMING] [KINE…
6873fb6 
…SIS] Updates to the Kinesis API

SPARK-6514 - Use correct region
SPARK-5960 - Allow AWS Credentials to be directly passed
SPARK-6656 - Specify kinesis application name explicitly
SPARK-7679 - Upgrade to latest KCL and AWS SDK.

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes apache#6147 from tdas/kinesis-api-update and squashes the following commits:

f23ea77 [Tathagata Das] Updated versions and updated APIs
373b201 [Tathagata Das] Updated Kinesis API
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
@tdas @jeanlyn
[SPARK-6514] [SPARK-5960] [SPARK-6656] [SPARK-7679] [STREAMING] [KINE…
bee688a 
…SIS] Updates to the Kinesis API

SPARK-6514 - Use correct region
SPARK-5960 - Allow AWS Credentials to be directly passed
SPARK-6656 - Specify kinesis application name explicitly
SPARK-7679 - Upgrade to latest KCL and AWS SDK.

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes apache#6147 from tdas/kinesis-api-update and squashes the following commits:

f23ea77 [Tathagata Das] Updated versions and updated APIs
373b201 [Tathagata Das] Updated Kinesis API
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@tdas @nemccarthy
[SPARK-6514] [SPARK-5960] [SPARK-6656] [SPARK-7679] [STREAMING] [KINE…
9e5196d 
…SIS] Updates to the Kinesis API

SPARK-6514 - Use correct region
SPARK-5960 - Allow AWS Credentials to be directly passed
SPARK-6656 - Specify kinesis application name explicitly
SPARK-7679 - Upgrade to latest KCL and AWS SDK.

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes apache#6147 from tdas/kinesis-api-update and squashes the following commits:

f23ea77 [Tathagata Das] Updated versions and updated APIs
373b201 [Tathagata Das] Updated Kinesis API
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@tdas @AmplabJenkins @SparkQA @cfregly