[SPARK-9780][Streaming][Kafka] prevent NPE if KafkaRDD instantiation …

[koeninger]

…fails

[SparkQA]

Test build #40652 has finished for PR 8133 at commit 406259d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tdas]

LGTM. Merging this to master and 1.5. Thanks!

[shyamsalimkumar]

I get the same NPE issue in Spark 1.4.1 also. I'm not sure what's wrong (yet) as my spark streaming job was running fine printing out Array[Byte] data, but after I added a custom decoder (Avro to my class Swipe) it keeps spitting the NPE error (consumer is null). Any hints on what could be the root cause?

val swipes = KafkaUtils.createDirectStream[Array[Byte], Array[Byte], DefaultDecoder, DefaultDecoder](ssc, kafkaParams, topics) works, but
val swipes = KafkaUtils.createDirectStream[Array[Byte], Swipe, DefaultDecoder, SwipeDecoder](ssc, kafkaParams, topics) doesn't. In that aspect the NPE is sorta misleading (maybe).

[koeninger]

KafkaRDD attempts to instantiate the value decoder right before connecting
the consumer:

val valueDecoder = classTag[T].runtimeClass.getConstructor(classOf[

VerifiableProperties])

  .newInstance(kc.config.props)

  .asInstanceOf[Decoder[V]]

val consumer = connectLeader

Something's wrong with your SwipeDecoder

On Mon, Aug 17, 2015 at 3:16 AM, Shyam S Kumar notifications@github.com
wrote:

I get the same NPE issue in Spark 1.4.1 also. I'm not sure what's wrong
(yet) as my spark streaming job was running fine printing out Array[Byte]
data, but after I added a custom decoder (Avro to my class Swipe) it
keeps spitting the NPE error (consumer is null). Any hints on what could
be the root cause?

val swipes = KafkaUtils.createDirectStream[Array[Byte], Array[Byte],
DefaultDecoder, DefaultDecoder](ssc, kafkaParams, topics) works, but
val swipes = KafkaUtils.createDirectStream[Array[Byte], Swipe,
DefaultDecoder, SwipeDecoder](ssc, kafkaParams, topics) doesn't. In that
aspect the NPE is sorta misleading (maybe).

—
Reply to this email directly or view it on GitHub
#8133 (comment).

[shyamsalimkumar]

Yes... I know. 😄 I had forgotten to accept props: VerifiableProperties in it. Silly me. But I still think that the error thrown is misleading (unless of course you know the above).

[koeninger]

This PR has already been merged to suppress the NPE when someone messes up
instantiation....
On Aug 18, 2015 1:13 AM, "Shyam S Kumar" notifications@github.com wrote:

Yes... I know. [image: 😄] I had forgotten to accept props:
VerifiableProperties in it. Silly me. But I still think that the error
thrown is misleading (unless of course you know the above).

—
Reply to this email directly or view it on GitHub
#8133 (comment).