ClassNotFoundException: org.apache.avro.message.BinaryMessageEncoder

[heuermh]

In downstream https://github.com/heuermh/adam-examples, which builds a fat jar using maven-shade-plugin, running on Spark 2.3.2 fails

$ spark-submit --class com.github.heuermh.adam.examples.CountAlignmentsPerRead target/adam-examples_2.11-0.24.0-SNAPSHOT.jar src/test/resources/small.sam
...
2018-11-15 10:37:10 INFO  ADAMContext:3279 - Loading src/test/resources/small.sam as BAM/CRAM/SAM and converting to AlignmentRecords.
2018-11-15 10:37:10 INFO  ADAMContext:1543 - Loaded header from file:/Users/heuermh/working/adam-examples/src/test/resources/small.sam
2018-11-15 10:37:10 INFO  MemoryStore:54 - Block broadcast_0 stored as values in memory (estimated size 290.6 KB, free 366.0 MB)
Exception in thread "main" java.lang.NoClassDefFoundError: Lorg/apache/avro/message/BinaryMessageEncoder;
	at java.lang.Class.getDeclaredFields0(Native Method)
	at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
	at java.lang.Class.getDeclaredField(Class.java:2068)
	at org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:240)
	at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:189)
	at org.apache.avro.specific.SpecificDatumReader.<init>(SpecificDatumReader.java:34)
	at org.bdgenomics.adam.serialization.AvroSerializer.<init>(ADAMKryoRegistrator.scala:43)
	at org.bdgenomics.adam.models.VariantContextSerializer.<init>(VariantContext.scala:94)
	at org.bdgenomics.adam.serialization.ADAMKryoRegistrator.registerClasses(ADAMKryoRegistrator.scala:179)
	at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$6.apply(KryoSerializer.scala:136)
	at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$6.apply(KryoSerializer.scala:136)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
	at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:136)
	at org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:324)
	at org.apache.spark.serializer.KryoSerializerInstance.<init>(KryoSerializer.scala:309)
	at org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:218)
	at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:288)
	at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:127)
	at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:88)
	at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
	at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
	at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1482)
	at org.apache.spark.rdd.NewHadoopRDD.<init>(NewHadoopRDD.scala:79)
	at org.apache.spark.SparkContext$$anonfun$newAPIHadoopFile$2.apply(SparkContext.scala:1154)
	at org.apache.spark.SparkContext$$anonfun$newAPIHadoopFile$2.apply(SparkContext.scala:1140)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.SparkContext.withScope(SparkContext.scala:693)
	at org.apache.spark.SparkContext.newAPIHadoopFile(SparkContext.scala:1140)
	at org.bdgenomics.adam.rdd.ADAMContext$$anonfun$loadBam$1.apply(ADAMContext.scala:1579)
	at org.bdgenomics.adam.rdd.ADAMContext$$anonfun$loadBam$1.apply(ADAMContext.scala:1522)
	at scala.Option.fold(Option.scala:158)
	at org.apache.spark.rdd.Timer.time(Timer.scala:48)
	at org.bdgenomics.adam.rdd.ADAMContext.loadBam(ADAMContext.scala:1522)
	at org.bdgenomics.adam.rdd.ADAMContext$$anonfun$loadAlignments$1.apply(ADAMContext.scala:3280)
	at org.bdgenomics.adam.rdd.ADAMContext$$anonfun$loadAlignments$1.apply(ADAMContext.scala:3271)
	at scala.Option.fold(Option.scala:158)
	at org.apache.spark.rdd.Timer.time(Timer.scala:48)
	at org.bdgenomics.adam.rdd.ADAMContext.loadAlignments(ADAMContext.scala:3271)
	at com.github.heuermh.adam.examples.CountAlignmentsPerRead$.main(CountAlignmentsPerRead.scala:44)
	at com.github.heuermh.adam.examples.CountAlignmentsPerRead.main(CountAlignmentsPerRead.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.avro.message.BinaryMessageEncoder
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 52 more

[ADDale]

I am getting the same error. Can you please confirm if this has been fixed? I am using avro 1..8.2 version.
Can we use some other version to avoid this issue.

[heuermh]

Hello @ADDale! This is a runtime classpath issue that shows up in a downstream project that extends from ADAM. What is the situation in which you run into this?