SPARK-2757 [BUILD] [STREAMING] Add Mima test for Spark Sink after 1.10 is released

[srowen]

Re-enable MiMa for Streaming Flume Sink module, now that 1.1.0 is released, per the JIRA TO-DO. That's pretty much all there is to this.

[SparkQA]

Test build #24893 has started for PR 3842 at commit 0e5ba5c.

• This patch merges cleanly.

[SparkQA]

Test build #24893 has finished for PR 3842 at commit 0e5ba5c.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24893/
Test FAILed.

[srowen]

Interesting situation. There is a MiMa failure since SPARK-3154 / bcb5cda changed a method after 1.2.0, but, it's private[sink]. I should just exclude this failure I believe.

[SparkQA]

Test build #24895 has started for PR 3842 at commit 50ff80e.

• This patch merges cleanly.

[SparkQA]

Test build #24895 has finished for PR 3842 at commit 50ff80e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24895/
Test PASSed.

[harishreedharan]

+1. Looks good to me!

[tdas]

Just wondering what is necessity for binary compatibility here? I am guessing that one should be able to operation Spark Streaming 1.2 programs with a deployed Sink from 1.1, right? For that what is API that needs to stay compatible? Isnt it only the Avro protocol? So we should figure out what are all the stuff that are needed make MIMA ignore the rest, isnt it?

[tdas]

A quick pass shows me that spark-streaming-flume uses the following from the spark-streaming-flume-sink module

• The generated Avro protocol classes - SparkFlumeProtcol, EventBatch, SparkSinkEvent
• SparkSinkUtils class

Should we consider adding all the other sink classes (TransactionProcessor, SparkAvroCallbackHandler, etc.) to Mima excludes right now? Or progressively add them as required?

[srowen]

@harishreedharan might know better; it was his suggestion to track this with MiMa. I suppose it can be enabled, to err on the side of tracking these things, and exclude/disable later when needed?

[harishreedharan]

I believe we need to track only the Avro classes and the SparkSink class
(not sure if we need binary compat for this either - I think API compat is
all we need for even the SparkSink class). Other than that we should be
fine since the jar that contains the sink itself has the other classes, so
binary compat isn't an issue.

On Tuesday, December 30, 2014, Sean Owen notifications@github.com wrote:

@harishreedharan https://github.com/harishreedharan might know better;
it was his suggestion to track this with MiMa. I suppose it can be enabled,
to err on the side of tracking these things, and exclude/disable later when
needed?

—
Reply to this email directly or view it on GitHub
#3842 (comment).

[tdas]

Okay @srowen lets do that - err on the side of tracking these things and disable later when needed. I am merging this.