Version upgrade to Scala 2.12.17, Spark 3.4.1, Delta 2.4.0.

[ivan-veselovsky]

No description provided.

[parisni]

many thanks for your work, one remark about commenting stuff, but overall LGTM

[parisni]

do you have some context for commenting this ?

[ivan-veselovsky]

Problem is that with updated Spark it becomes impossible to have in Dataset columns with types Array(Option(X)), where X is java.sql.Date, java.sql.Timestamp, scal.math.BigDecimal. E.g the following code fragment

Seq(Array(Some(new Date(1L)))).toDS()

causes error org.apache.spark.SparkRuntimeException: Error while encoding: java.lang.RuntimeException: scala.Some is not a valid external type for schema of date .

I commented the problematic lines, because, to be honest, I'm not sure how to fix this properly.
Simplest solution would be to remove these commented lines.
Any suggestions?

[ivan-veselovsky]

Removed commented code.

[ivan-veselovsky]

many thanks for your work, one remark about commenting stuff, but overall LGTM

We use this project in our work, so thank you!

One more question: do you have any plans to release this library onto Maven Central ?

[parisni]

One more question: do you have any plans to release this library onto Maven Central ?

That would be great. Not sure every module has value. Which feature are you using so far ?

…

On September 8, 2023 12:10:55 PM UTC, Ivan ***@***.***> wrote: > many thanks for your work, one remark about commenting stuff, but overall LGTM We use this project in our work, so thank you! One more question: do you have any plans to release this library onto Maven Central ? -- Reply to this email directly or view it on GitHub: #35 (comment) You are receiving this because you commented. Message ID: ***@***.***>

[ivan-veselovsky]

One more question: do you have any plans to release this library onto Maven Central ?
That would be great. Not sure every module has value. Which feature are you using so far ?
…
On September 8, 2023 12:10:55 PM UTC, Ivan @.> wrote: > many thanks for your work, one remark about commenting stuff, but overall LGTM We use this project in our work, so thank you! One more question: do you have any plans to release this library onto Maven Central ? -- Reply to this email directly or view it on GitHub: #35 (comment) You are receiving this because you commented. Message ID: @.>

So far we use spark-postgres (depends on spark-dataframe).

[parisni]

LGTM thanks a lot

[ivan-veselovsky]

Super! Could we merge it?

[parisni]

sadly not. maybe @Dubrzr can ?

[ivan-veselovsky]

Hello, guys, @parisni, @Dubrzr , can anybody please merge this PR?