Problem using this library with Apache Spark

[melrief]

I'm trying to use this library and it works very well except when I use it in a Spark (both 1.3.1 and 1.4.0) application. I get an IllegalAccessError on the LogstashBasicMarker but I'm not sure why:

Exception in thread "main" java.lang.IllegalAccessError: tried to access method org.slf4j.helpers.BasicMarker.<init>(Ljava/lang/String;)V from class org.slf4j.helpers.LogstashBasicMarker
    at org.slf4j.helpers.LogstashBasicMarker.<init>(LogstashBasicMarker.java:25)
    at net.logstash.logback.marker.LogstashMarker.<init>(LogstashMarker.java:35)
    at net.logstash.logback.marker.SingleFieldAppendingMarker.<init>(SingleFieldAppendingMarker.java:37)
    at net.logstash.logback.marker.ObjectAppendingMarker.<init>(ObjectAppendingMarker.java:47)
    at net.logstash.logback.marker.Markers.append(Markers.java:59)
        ...

Maybe LogstashBasicMarker should not be used to bypass the visibility level of the constructor of BasicMarker? Anyway, do you know how to solve this problem?

[philsttr]

I don't have any experience with spark. If you can provide a small sample application that reproduces the problem, I'll be happy to look into it.

[melrief]

I created a small example that just creates a Marker. To see the error, you can create the assembly of the project with sbt assembly and then run it as Spark job with something like ${SPARK_HOME}/bin/spark-submit --master ${MASTER} logbackSparkProblem-assembly-1.0.jar (based on when you have spark installed, if it's a local install then you can set MASTER="local[2]").

[philsttr]

Thanks for the example @melrief! It's a good thing you provided it, because I thought you were referring to the spark framework.

I have reproduced the problem using apache spark.

By default, org.slf4j.helpers.BasicMarker is loaded from the spark assembly jar (rather than your assembly). You can see this by using verbose classloader output like this:

SPARK_SUBMIT_OPTS="-verbose:class" ${SPARK_HOME}/bin/spark-submit --master ${MASTER} target/scala-2.11/logbackSparkProblem-assembly-1.0.jar

which outputs:

[Loaded org.slf4j.helpers.BasicMarker from file:/tmp/spark-1.4.0-bin-hadoop2.6/lib/spark-assembly-1.4.0-hadoop2.6.0.jar]
[Loaded org.slf4j.helpers.LogstashBasicMarker from file:/tmp/logbackSparkProblem/target/scala-2.11/logbackSparkProblem-assembly-1.0.jar]

A workaround is to set spark.driver.userClassPathFirst=true in your spark config (either via --conf spark.driver.userClassPathFirst=true OR the spark-defaults.conf file). This forces spark to load BasicMarker from your assembly. You can see this by running:

SPARK_SUBMIT_OPTS="-verbose:class" ${SPARK_HOME}/bin/spark-submit --conf spark.driver.userClassPathFirst=true --master ${MASTER} target/scala-2.11/logbackSparkProblem-assembly-1.0.jar

which outputs

[Loaded org.slf4j.helpers.BasicMarker from file:/tmp/logbackSparkProblem/target/scala-2.11/logbackSparkProblem-assembly-1.0.jar]
[Loaded org.slf4j.helpers.LogstashBasicMarker from file:/tmp/logbackSparkProblem/target/scala-2.11/logbackSparkProblem-assembly-1.0.jar]

I'm not sure why spark prevents access to the BasicMarker package-private constructor. I haven't investigated the classloader configuration being in spark yet. Normally, this will work fine in a typical java application.

Other than the workaround I mentioned above, I believe the only fix for this will be to either:
A. Modify BasicMarker in slf4j-api, to expose a public constructor
OR
B. Remove usage of org.slf4j.helpers.BasicMarker and reimplement the logic from it in logstash-logback-encoder

I'll implement B for the time being.

[melrief]

Sounds good to me and thanks for the workarounds!

[melrief]

I've created the PR #108 for this that reimplements the logic of BasicMarker into LogstashBasicMarker

[philsttr]

Thanks for the PR!

[philsttr]

4.4 has been released with this fix