NoSuchMethodError due to kryo minor-version mismatch

[ryan-williams]

Not sure we want to or should do anything about this, but documenting it fwiw.

I have some code that depends on adam-core_2.10-0.18.2, and from that picks up a dependency on Kryo 2.24. Meanwhile, spark-core 1.6.0 depends on Chill 0.5.0 which depends on Kryo 2.21.

My code calls Input.total which is a long in Kryo 2.24 but an int in Kryo 2.21, so I get a NoSuchMethodError because my code was compiled against 2.24 but picks up Spark's 2.21 at runtime.

I'm going to look into whether this can be mitigated by one of these "user classpath first" options that I've seen discussed around Spark-land.

[heuermh]

At a quick glance it doesn't appear we declare a runtime or compile time dependency on kyro, even though we define a version for it in the <dependencyManagement> section. Maybe adding a runtime scope to your pom or to say adam-cli would override the Spark runtime dependency version?

[fnothaft]

Maybe adding a runtime scope to your pom or to say adam-cli would override the Spark runtime dependency version?

IIRC, it does, but you need to muck around with classpaths in Spark submit, or something of that sort. It's a bit ugly of a bug. I ran into this or something similar on a pre-1.6 custom build.

[heuermh]

Is there anything actionable on this for the 0.20.0 release?

[fnothaft]

I don't think there's anything actionable for this on 0.23.0. I'm going to close this as won't fix. @ryan-williams please reopen if you think I am in error.