Limitations on the output size?

[rks0191]

Well I am running into a bit of an issue with this DataLineageWriter.

I have a custom class which has multiple operations and the table size is huge(around 40 different columns)
I wanted to check the Lineage Json to understand something.
So what I ended up doing was something like this

class A extends DataLineageWriter{
override def store(lineage:DataLineage)(implicit ec:ExecutionContext)={
Future{
val logger = CustomLogger().getInstance()

}.recoverWith{
case e:Exception => println("caught an exception")
Future.successful("Error")

}
}
}

However to make this work, I need to increase spark.debug.maxToStringFields to 30
However post the same Iam still unable to get to the Custom Writer here. Any pointers would be really helpful

Note: Iam still using spline 0.3

[wajda]

hi @rks0191,
Not sure I understand what exactly is not working for you? Can you clarify it please?
Also note that Spline 0.3 is old and isn't supported.

[rks0191]

HI @wajda .
Sorry for being a bit unclear earlier
I was trying to Do a lineaging on my Spark Application. This Spark Application has Multiple transformation which Iam interested on. So what i Did was in between my Spark application I added in

SparkLineageInitializer.enableLineageTracking(spark)

Now the way I understand the Spline Library, If i need to persist the logs, then I need to and set spline.persistence.factory to the custom logger if Iam logging things into.

So My Custom Logger extends DataLineageWriter and has the store method which could deserialize the object and write the same to the File

So here are my questions

1: Despite of enabling the spline persistence factory,The Store method is not called in My Custom Logger. So is there a limit on the amount of data that could be logged?
2: Does toDebugString property of SparkConf have any impact in generating a lineage tree?

[wajda]

There's no specific limit of any kind, and it doesn't depend on toDebugString.
There could be two reasons why your custom writer isn't called:

1. Spline doesn't see your custom config property.
   Double check your configuration - https://absaoss.github.io/spline/0.3.html#configuration
   Make sure it's actually respected by Spline (for example set a non-existing class name and see if Spline fails with class not found).
2. The write command that your Spark job executes isn't supported by the Spline version you are using. If I remember correctly Spline 0.3 only supports writes to a file (DataFrame.write)

[pshaw17]

Hi @wajda
I'm using spline 0.7.4 version and I have multiple operations and classes so the output structure looks quite large and complex.
Is there any way to simplify structure .

[wajda]

Sorry I do not understand your question. The output lineage JSON is large and complex because as you said your execution plan is complex. What and how you would like to simplify?

[pshaw17]

okay so referring with an example image , we can see the lineage structure in the image but in my case I have multiple classes and the lineage structure generated for that is huge so is there any way we can simply it .

[wajda]

Oh, you mean to make the graph on the UI more compact? Well, currently you can't. But we'll think about it for the future. The biggest question here is what would be the criteria for grouping the nodes.