String length (...) exceeds the maximum length (20000000)

[sergiimk]

After migration to Spark 3.5 we pulled in new Jackson version that introduces a default limit on the length of string (20MB).

See: FasterXML/jackson-core#1014

This creates an issue for some of our examples operating with very large GeoJSON fields.

Neither the jackson library nor Spark expose it as configuration option, so the only way to fix it is to fork Spark, which we won't be doing.

This ticket is to take note of the issue, but most likely we will solve it by switching away from Livy and towards something like Spark Connect that transfers data in Arrow format.

[sergiimk]

Reported an upstream issue: https://issues.apache.org/jira/browse/SPARK-47150

[CrzEP]

so.anyone can tell me ,how to let maxLenLim change to more bigger?

[sergiimk]

We have not found a workaround. I think the only solution is to patch Spark code ... which is a lot of work.

We are only hitting this issue when querying large GIS datasets that have geometry properties within them, so we are waiting for upstream issue to be fixed.

I suggest leaving a comment on Spark's Jira.

Meanwhile our strategy is to depend less and less on Spark in favor of other engines.

[sergiimk]

We went the hard way and patched default configuration directly in Livy.

Commit:
kamu-data/incubator-livy@d1c898c

Released in engine image version: v0.23.1