log4j error in spark 3.5.0 (stackable version 23.11.0) #330

paulpaul1076 · 2023-12-22T06:49:58Z

paulpaul1076
Dec 22, 2023

My spark jobs work, but I get an exception in every job that I run with the following stack trace:

ERROR StatusConsoleListener Could not create plugin of type class org.apache.logging.log4j.core.layout.XmlLayout for element XMLLayout: java.lang.NoClassDefFoundError: com/fasterxml/jackson/dataformat/xml/XmlMapper
 java.lang.NoClassDefFoundError: com/fasterxml/jackson/dataformat/xml/XmlMapper
        at java.base/java.lang.ClassLoader.defineClass1(Native Method)
        at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1022)
        at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
        at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
        at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
        at org.apache.logging.log4j.core.layout.XmlLayout.<init>(XmlLayout.java:97)
        at org.apache.logging.log4j.core.layout.XmlLayout.<init>(XmlLayout.java:63)
        at org.apache.logging.log4j.core.layout.XmlLayout$Builder.build(XmlLayout.java:78)
        at org.apache.logging.log4j.core.layout.XmlLayout$Builder.build(XmlLayout.java:67)
        at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:124)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1138)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1063)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1055)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1055)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:664)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:258)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:304)
        at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:621)
        at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:694)
        at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:711)
        at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
        at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:155)
        at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)
        at org.apache.logging.log4j.LogManager.getContext(LogManager.java:309)
        at org.apache.log4j.Hierarchy$PrivateLogManager.getContext(Hierarchy.java:86)
        at org.apache.log4j.Hierarchy.getContext(Hierarchy.java:99)
        at org.apache.log4j.Category.<init>(Category.java:188)
        at org.apache.log4j.Logger.<init>(Logger.java:57)
        at org.apache.log4j.spi.RootLogger.<init>(RootLogger.java:39)
        at org.apache.log4j.LogManager.<clinit>(LogManager.java:72)
        at org.apache.log4j.Logger.getLogger(Logger.java:40)
        at com.amazonaws.thirdparty.apache.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:262)
        at com.amazonaws.thirdparty.apache.logging.impl.Log4JLogger.<init>(Log4JLogger.java:108)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at com.amazonaws.thirdparty.apache.logging.impl.LogFactoryImpl.createLogFromClass(LogFactoryImpl.java:1025)
        at com.amazonaws.thirdparty.apache.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:844)
        at com.amazonaws.thirdparty.apache.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:541)
        at com.amazonaws.thirdparty.apache.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:292)
        at com.amazonaws.thirdparty.apache.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:269)
        at com.amazonaws.thirdparty.apache.logging.LogFactory.getLog(LogFactory.java:657)
        at com.amazonaws.ClientConfiguration.<clinit>(ClientConfiguration.java:47)
        at org.apache.hadoop.fs.s3a.S3AUtils.createAwsConf(S3AUtils.java:1258)
        at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:114)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:898)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:534)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)
        at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
        at org.apache.spark.util.DependencyUtils$.resolveGlobPath(DependencyUtils.scala:317)
        at org.apache.spark.util.DependencyUtils$.$anonfun$resolveGlobPaths$2(DependencyUtils.scala:273)
        at org.apache.spark.util.DependencyUtils$.$anonfun$resolveGlobPaths$2$adapted(DependencyUtils.scala:271)
        at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
        at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
        at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
        at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
        at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
        at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
        at org.apache.spark.util.DependencyUtils$.resolveGlobPaths(DependencyUtils.scala:271)
        at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$4(SparkSubmit.scala:392)
        at scala.Option.map(Option.scala:230)
        at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:392)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:964)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.dataformat.xml.XmlMapper
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
        ... 78 more
ERROR StatusConsoleListener Unable to invoke factory method in class org.apache.logging.log4j.core.layout.XmlLayout for element XMLLayout: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.layout.XmlLayout
 java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.layout.XmlLayout
        at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:260)
        at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:136)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1138)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1063)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1055)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1055)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:664)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:258)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:304)
        at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:621)
        at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:694)
        at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:711)
        at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
        at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:155)
        at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)
        at org.apache.logging.log4j.LogManager.getContext(LogManager.java:309)
        at org.apache.log4j.Hierarchy$PrivateLogManager.getContext(Hierarchy.java:86)
        at org.apache.log4j.Hierarchy.getContext(Hierarchy.java:99)
        at org.apache.log4j.Category.<init>(Category.java:188)
        at org.apache.log4j.Logger.<init>(Logger.java:57)
        at org.apache.log4j.spi.RootLogger.<init>(RootLogger.java:39)
        at org.apache.log4j.LogManager.<clinit>(LogManager.java:72)
        at org.apache.log4j.Logger.getLogger(Logger.java:40)
        at com.amazonaws.thirdparty.apache.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:262)
        at com.amazonaws.thirdparty.apache.logging.impl.Log4JLogger.<init>(Log4JLogger.java:108)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at com.amazonaws.thirdparty.apache.logging.impl.LogFactoryImpl.createLogFromClass(LogFactoryImpl.java:1025)
        at com.amazonaws.thirdparty.apache.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:844)
        at com.amazonaws.thirdparty.apache.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:541)
        at com.amazonaws.thirdparty.apache.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:292)
        at com.amazonaws.thirdparty.apache.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:269)
        at com.amazonaws.thirdparty.apache.logging.LogFactory.getLog(LogFactory.java:657)
        at com.amazonaws.ClientConfiguration.<clinit>(ClientConfiguration.java:47)
        at org.apache.hadoop.fs.s3a.S3AUtils.createAwsConf(S3AUtils.java:1258)
        at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:114)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:898)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:534)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)
        at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
        at org.apache.spark.util.DependencyUtils$.resolveGlobPath(DependencyUtils.scala:317)
        at org.apache.spark.util.DependencyUtils$.$anonfun$resolveGlobPaths$2(DependencyUtils.scala:273)
        at org.apache.spark.util.DependencyUtils$.$anonfun$resolveGlobPaths$2$adapted(DependencyUtils.scala:271)
        at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
        at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
        at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
        at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
        at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
        at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
        at org.apache.spark.util.DependencyUtils$.resolveGlobPaths(DependencyUtils.scala:271)
        at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$4(SparkSubmit.scala:392)
        at scala.Option.map(Option.scala:230)
        at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:392)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:964)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

How do I make this exception go away?

paulpaul1076 · 2023-12-22T07:04:57Z

paulpaul1076
Dec 22, 2023
Author

I solved it by adding this to extra classs path:

    spark.driver.extraClassPath: /stackable/spark/extra-jars/*
    spark.executor.extraClassPath: /stackable/spark/extra-jars/*

This folder inside the spark docker image has the jackson xml dependency.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

log4j error in spark 3.5.0 (stackable version 23.11.0) #330

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment

{{title}}

Select a reply

log4j error in spark 3.5.0 (stackable version 23.11.0) #330

paulpaul1076 Dec 22, 2023

Replies: 1 comment

paulpaul1076 Dec 22, 2023 Author

paulpaul1076
Dec 22, 2023

paulpaul1076
Dec 22, 2023
Author