Facing issues with spline 2.12

[uday1409]

Describe the bug

Lineage is not captured and throwing error that spark context is stopped. below is the error. Can you please help me where it is going wrong

20/09/23 03:13:58 ERROR DatabricksMain$DBUncaughtExceptionHandler: Uncaught exception in thread spark-listener-group-shared!
java.lang.NoSuchMethodError: org.json4s.Formats.$plus$plus(Lscala/collection/Traversable;)Lorg/json4s/Formats;
	at za.co.absa.commons.json.format.JavaTypesSupport.formats(JavaTypesSupport.scala:23)
	at za.co.absa.commons.json.format.JavaTypesSupport.formats$(JavaTypesSupport.scala:23)
	at __wrapper$1$b519b81e4ed7491ba31ccb47bdf0c79c.
__wrapper$1$b519b81e4ed7491ba31ccb47bdf0c79c$$anon$1.formats(no source file)
	at za.co.absa.commons.json.AbstractJsonSerDe.$init$(AbstractJsonSerDe.scala:32)
	at __wrapper$1$b519b81e4ed7491ba31ccb47bdf0c79c.
__wrapper$1$b519b81e4ed7491ba31ccb47bdf0c79c$$anon$1.<init>(no source file)
	at __wrapper$1$b519b81e4ed7491ba31ccb47bdf0c79c.
__wrapper$1$b519b81e4ed7491ba31ccb47bdf0c79c$.$anonfun$wrapper$1 (no source file)
	at za.co.absa.spline.harvester.json.HarvesterJsonSerDe$.<init>(HarvesterJsonSerDe.scala:37)
	at za.co.absa.spline.harvester.json.HarvesterJsonSerDe$.<clinit>(HarvesterJsonSerDe.scala)

Versions

2.12

[wajda]

Seems like json4s library incompatibility.
Are you deploying the Spline agent as a pre-built bundle or you create your own assembly?
What Spark version are you using?

[uday1409]

We are using as bundle and trying to run it on Spark3.0. I suspected that it could be because of package incompatibility, and even tried with newer version of json4s but it did not work.

[wajda]

Yes, Spark 3.0 isn't supported yet, partially due to some libraries incompatibilities like this one.
We have tickets for that:

• Spark 3.0 support #93
• Json4s: Ensure byte code compatibility with Json4s 3.6.7 commons#27