[SPARK-6087][CORE] Provide actionable exception if Kryo buffer is not large enough

[levkhomich]

A simple try-catch wrapping KryoException to be more informative.

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

ok to test

[srowen]

LGTM.

[SparkQA]

Test build #28388 has started for PR 4947 at commit 48ab7f9.

• This patch merges cleanly.

[SparkQA]

Test build #28388 has finished for PR 4947 at commit 48ab7f9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28388/
Test PASSed.

[viirya]

Original message (Available and requested size) in KryoException is useful too. Is it better to include original message too?

[levkhomich]

Original exception is preserved as cause, so it is printed anyway.

[viirya]

But as the Exception's Constructor Detail (http://docs.oracle.com/javase/7/docs/api/java/lang/Exception.html#Exception(java.lang.String,%20java.lang.Throwable) states,

Note that the detail message associated with cause is not automatically incorporated in this exception's detail message.

Is it sure that it will be printed?

[srowen]

The cause stack trace / message would be printed by printStackTrace. It would not become part of the message from this new SparkException. Net-net I think it wouldn't hurt to just add additional info to the new SparkException message if it's deemed useful.

[levkhomich]

Sure, you can check example of stack trace here.

[levkhomich]

@srowen @viirya I've squashed corresponding change.

[SparkQA]

Test build #28397 has started for PR 4947 at commit 0f7a947.

• This patch merges cleanly.

[srowen]

LGTM. I'll wait a bit longer for more comments.

[SparkQA]

Test build #28397 has finished for PR 4947 at commit 0f7a947.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class BinaryClassificationMetrics(JavaModelWrapper):

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28397/
Test PASSed.

[sryza]

Is this not needed for serializeStream as well?

[levkhomich]

@sryza KryoSerializer.maxBufferSize is used only in KryoSerializer.newKryoOutput, which, in turn, is used only in KryoSerializerInstance. KryoSerializationStream uses output passed in arguments, so spark.kryoserializer.buffer.max.mb doesn't affect it. Same applies to JavaIterableWrapperSerializer.

[srowen]

Yes, that looks correct. In the other cases, we aren't writing to an internal buffer, not one that is controlled by this buffer size setting.