Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tried to access method org.apache.avro.specific.SpecificData.<init>()V from class ProcessingStep #2151

Closed
heuermh opened this issue May 20, 2019 · 4 comments
Closed

Tried to access method org.apache.avro.specific.SpecificData.<init>()V from class ProcessingStep #2151

heuermh opened this issue May 20, 2019 · 4 comments
Milestone
0.27.0

Comments

@heuermh
Copy link
Member

heuermh commented May 20, 2019 •
edited
Loading

Error thrown in Jenkins test

https://amplab.cs.berkeley.edu/jenkins/job/ADAM-prb/2998/HADOOP_VERSION=2.7.5,SCALAVER=2.12,SPARK_VERSION=2.4.3,label=ubuntu/console

19/05/20 09:49:00 ERROR rdd.ADAMContext: Loading failed for file:/tmp/adamTesthzf2rx2/deleteMePleaseThisIsNoLongerNeeded/mouse_chrM.bam:
tried to access method org.apache.avro.specific.SpecificData.<init>()V from class org.bdgenomics.formats.avro.ProcessingStep
	org.bdgenomics.formats.avro.ProcessingStep.<clinit>(ProcessingStep.java:21)
	org.bdgenomics.adam.rdd.ADAMContext$.convertSAMProgramRecord(ADAMContext.scala:1074)
	org.bdgenomics.adam.rdd.ADAMContext.$anonfun$loadBamPrograms$1(ADAMContext.scala:1150)
	scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
	scala.collection.Iterator.foreach(Iterator.scala:941)
	scala.collection.Iterator.foreach$(Iterator.scala:941)
	scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
	scala.collection.IterableLike.foreach(IterableLike.scala:74)
	scala.collection.IterableLike.foreach$(IterableLike.scala:73)
	scala.collection.AbstractIterable.foreach(Iterable.scala:56)
	scala.collection.TraversableLike.map(TraversableLike.scala:237)
	scala.collection.TraversableLike.map$(TraversableLike.scala:230)
	scala.collection.AbstractTraversable.map(Traversable.scala:108)
	org.bdgenomics.adam.rdd.ADAMContext.loadBamPrograms(ADAMContext.scala:1150)
	org.bdgenomics.adam.rdd.ADAMContext.$anonfun$loadBam$4(ADAMContext.scala:1547)
	scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:244)
	scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
	scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
	scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
	scala.collection.TraversableLike.flatMap(TraversableLike.scala:244)
	scala.collection.TraversableLike.flatMap$(TraversableLike.scala:241)
	scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:198)
	org.bdgenomics.adam.rdd.ADAMContext.$anonfun$loadBam$1(ADAMContext.scala:1537)
	scala.Option.fold(Option.scala:175)
	org.apache.spark.rdd.Timer.time(Timer.scala:48)
19/05/20 09:49:00 INFO memory.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 295.6 KB, free 366.0 MB)
Command body threw exception:
java.lang.NoSuchMethodError: org.apache.avro.Schema$Parser.parse(Ljava/lang/String;[Ljava/lang/String;)Lorg/apache/avro/Schema;
19/05/20 09:49:00 INFO cli.TransformAlignments: Overall Duration: 2.51 secs
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.avro.Schema$Parser.parse(Ljava/lang/String;[Ljava/lang/String;)Lorg/apache/avro/Schema;
	at org.bdgenomics.formats.avro.Variant.<clinit>(Variant.java:18)
	at sun.misc.Unsafe.ensureClassInitialized(Native Method)
	at sun.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(UnsafeFieldAccessorFactory.java:43)
	at sun.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:156)
	at java.lang.reflect.Field.acquireFieldAccessor(Field.java:1088)
	at java.lang.reflect.Field.getFieldAccessor(Field.java:1069)
	at java.lang.reflect.Field.get(Field.java:393)
	at org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:205)
	at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:154)
	at org.apache.avro.specific.SpecificDatumReader.<init>(SpecificDatumReader.java:32)
	at org.bdgenomics.adam.serialization.AvroSerializer.<init>(ADAMKryoRegistrator.scala:43)
	at org.bdgenomics.adam.models.VariantContextSerializer.<init>(VariantContext.scala:94)
	at org.bdgenomics.adam.serialization.ADAMKryoRegistrator.registerClasses(ADAMKryoRegistrator.scala:190)
	at org.apache.spark.serializer.KryoSerializer.$anonfun$newKryo$7(KryoSerializer.scala:136)
	at org.apache.spark.serializer.KryoSerializer.$anonfun$newKryo$7$adapted(KryoSerializer.scala:136)
	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
	at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
	at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:136)
	at org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:324)
	at org.apache.spark.serializer.KryoSerializerInstance.<init>(KryoSerializer.scala:309)
	at org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:218)
	at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:288)
	at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:127)
	at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:88)
	at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
	at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
	at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1489)
	at org.apache.spark.rdd.NewHadoopRDD.<init>(NewHadoopRDD.scala:79)
	at org.apache.spark.SparkContext.$anonfun$newAPIHadoopFile$2(SparkContext.scala:1160)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.SparkContext.withScope(SparkContext.scala:699)
	at org.apache.spark.SparkContext.newAPIHadoopFile(SparkContext.scala:1146)
	at org.bdgenomics.adam.rdd.ADAMContext.$anonfun$loadBam$1(ADAMContext.scala:1583)
	at scala.Option.fold(Option.scala:175)
	at org.apache.spark.rdd.Timer.time(Timer.scala:48)
	at org.bdgenomics.adam.rdd.ADAMContext.loadBam(ADAMContext.scala:1523)
	at org.bdgenomics.adam.rdd.ADAMContext.$anonfun$loadAlignments$1(ADAMContext.scala:3282)
	at scala.Option.fold(Option.scala:175)
	at org.apache.spark.rdd.Timer.time(Timer.scala:48)
	at org.bdgenomics.adam.rdd.ADAMContext.loadAlignments(ADAMContext.scala:3273)
	at org.bdgenomics.adam.cli.TransformAlignments.run(TransformAlignments.scala:510)
	at org.bdgenomics.utils.cli.BDGSparkCommand.run(BDGCommand.scala:55)
	at org.bdgenomics.utils.cli.BDGSparkCommand.run$(BDGCommand.scala:46)
	at org.bdgenomics.adam.cli.TransformAlignments.run(TransformAlignments.scala:154)
	at org.bdgenomics.adam.cli.ADAMMain.apply(ADAMMain.scala:126)
	at org.bdgenomics.adam.cli.ADAMMain$.main(ADAMMain.scala:65)
	at org.bdgenomics.adam.cli.ADAMMain.main(ADAMMain.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Issue filed upstream https://issues.apache.org/jira/browse/SPARK-27781
@heuermh
Copy link
Member Author

heuermh commented May 20, 2019

Can reproduce with

$ export SPARK_DIST_CLASSPATH=$(~/bin/hadoop-2.7.7/bin/hadoop classpath)
$ spark-submit --version
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.3
      /_/
                        
Using Scala version 2.12.8, OpenJDK 64-Bit Server VM, 11.0.1
Branch 
Compiled by user  on 2019-05-01T05:46:39Z
Revision 
Url 
Type --help for more information.

$ ./bin/adam-submit --version
Using ADAM_MAIN=org.bdgenomics.adam.cli.ADAMMain
Using spark-submit=~/bin/spark-2.4.3-bin-without-hadoop-scala-2.12/bin/spark-submit

       e        888~-_         e            e    e
      d8b       888   \       d8b          d8b  d8b
     /Y88b      888    |     /Y88b        d888bdY88b
    /  Y88b     888    |    /  Y88b      / Y88Y Y888b
   /____Y88b    888   /    /____Y88b    /   YY   Y888b
  /      Y88b   888_-~    /      Y88b  /          Y888b

ADAM version: 0.27.0-SNAPSHOT
Commit: dd49f66cdc01640d2baf1635c92ca4f44190dd44 Build: 2019-05-20
Built for: Apache Spark 2.4.3, Scala 2.12.8, and Hadoop 2.7.5

$ ./bin/adam-submit transformAlignments adam-core/src/test/resources/small.sam small.adam

@heuermh
Copy link
Member Author

heuermh commented May 20, 2019 •
edited
Loading 

$ find spark-2.4.3-bin-hadoop2.7 *.jar | grep avro
jars/avro-1.8.2.jar
jars/avro-mapred-1.8.2-hadoop2.jar
jars/avro-ipc-1.8.2.jar

$ find spark-2.4.3-bin-without-hadoop-scala-2.12 *.jar | grep avro
jars/avro-mapred-1.8.2-hadoop2.jar

$ find hadoop-2.7.7 -name *.jar | grep avro
share/hadoop/mapreduce/lib/avro-1.7.4.jar
share/hadoop/kms/tomcat/webapps/kms/WEB-INF/lib/avro-1.7.4.jar
share/hadoop/tools/lib/avro-1.7.4.jar
share/hadoop/common/lib/avro-1.7.4.jar
hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/avro-1.7.4.jar

@heuermh heuermh added this to the 0.27.0 milestone May 21, 2019
@heuermh
Copy link
Member Author

heuermh commented May 21, 2019

Fixed by removing all version 1.7.4 avro jars from the Hadoop 2.7.7 binary installation.

See upstream issue and dev@spark mailing list thread for further discussion.

@heuermh heuermh closed this as completed May 21, 2019
@heuermh heuermh reopened this Sep 11, 2019
@heuermh
Copy link
Member Author

heuermh commented Sep 11, 2019

Fixed by #2206

@heuermh heuermh closed this as completed Sep 11, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
0.27.0
Development

No branches or pull requests
1 participant
@heuermh