[STREAMING] SPARK-1729. Make Flume pull data from source, rather than the current pu...

[harishreedharan]

...sh model

Currently Spark uses Flume's internal Avro Protocol to ingest data from Flume. If the executor running the
receiver fails, it currently has to be restarted on the same node to be able to receive data.

This commit adds a new Sink which can be deployed to a Flume agent. This sink can be polled by a new
DStream that is also included in this commit. This model ensures that data can be pulled into Spark from
Flume even if the receiver is restarted on a new node. This also allows the receiver to receive data on
multiple threads for better performance.

[AmplabJenkins]

Can one of the admins verify this patch?

[rxin]

@pwendell @tdas

[rxin]

Jenkins, test this please.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15074/

[harishreedharan]

Not sure why the build failed with a dependency resolution issue. It seems to work locally when I do sbt assembly locally. Does the order of modules specified in the build spec matter?

Any advice from someone more familiar with sbt?

On May 18, 2014, at 10:19 PM, UCB AMPLab notifications@github.com wrote:

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15074/

—
Reply to this email directly or view it on GitHub.

[srowen]

Pardon for jumping in with comments, but why a new module instead of external/flume?

[harishreedharan]

The sink will be deployed to a Flume agent and not within the spark application. Adding it in external/flume would require that all of the spark dependencies be bundled with the jar, while keeping this module separate (which does not depend on the rest of Spark) allows the user to simply deploy this jar to the Flume plugins directory. In fact, this module does not have any dependencies that Flume already does not pull in by default.

[harishreedharan]

Thanks for the comments @srowen!

[harishreedharan]

The latest commit should fix the build issue.

[harishreedharan]

I am going to update this to support polling from multiple Flume agents rather than just one.

[harishreedharan]

The latest commit adds support for polling several flume agents.

[tdas]

connections could be built in a more Scala-like functional style. Something like this.

connections = addresses.map { host =>  .....   new FlumeConnection(....) }.toArray

No need for ArrayBuilder

[harishreedharan]

This ends up in data being copied, but since it is one-time I guess it is ok.

[SparkQA]

QA results for PR 807:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class SparkSink extends AbstractSink with Logging with Configurable {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17298/consoleFull

[tdas]

The binary compatibility test failed because it tried to compare current version of flume-sink with previous version (which does not exist). So we need to make an exclusion here. Let me figure this out with @pwendell

[harishreedharan]

Thanks @tdas! I was trying to figure it out when I saw the failure - but I can't see a place to add the exclusions.

[harishreedharan]

Added sparkSink to mima excludes. This should fix the Jenkins failure.

[SparkQA]

QA tests have started for PR 807. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17303/consoleFull

[tdas]

Yeah, I will merge this as soon as this passes.

[SparkQA]

QA results for PR 807:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class SparkSink extends AbstractSink with Logging with Configurable {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17303/consoleFull

[SparkQA]

QA tests have started for PR 807. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17313/consoleFull

[tdas]

All right! Merging this!
THIS WAS A LOT OF WORK. THANK YOU VERY MUCH!

[SparkQA]

QA results for PR 807:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class SparkSink extends AbstractSink with Logging with Configurable {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17313/consoleFull

[SparkQA]

QA tests have started for PR 807. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17317/consoleFull

[SparkQA]

QA results for PR 807:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class SparkSink extends AbstractSink with Logging with Configurable {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17317/consoleFull

[SparkQA]

QA tests have started for PR 807. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17332/consoleFull

[SparkQA]

QA results for PR 807:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class SparkSink extends AbstractSink with Logging with Configurable {
class SparkSQLOperationManager(hiveContext: HiveContext) extends OperationManager with Logging {
class HadoopTableReader(

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17332/consoleFull

[harishreedharan]

SparkSQLOperationManager and HadoopTableReader were added to ASF git master. Since the github mirror is out of sync, the merge commit between this PR and that one makes the Jenkins script think that this PR added these two classes.