[SPARK-54571][CORE][SQL] Use LZ4 safeDecompressor

[pan3793]

What changes were proposed in this pull request?

Previously, lz4-java was upgraded to 1.10.1 to address CVEs,

• [SPARK-54597][BUILD] Upgrade lz4-java to 1.10.0 #53327
• [SPARK-54611][BUILD] Upgrade lz4-java to 1.10.1 #53347

while this casues significant performance drop, see the benchmark report at

• [SPARK-54693][CORE][TESTS] Add LZ4TPCDSDataBenchmark #53453

this PR follows the suggestion to migrate to safeDecompressor.

This PR also upgrades DB2 JDBC driver to a special version 12.1.3.0_special_74723 provided by DB2 team that bundles lz4-java 1.10.1 which addresses NoSuchMethodError issue (it only affects test).

Why are the changes needed?

Mitigate performance regression.

Does this PR introduce any user-facing change?

No, except for performance.

How was this patch tested?

GHA for functionality, benchmark for performance.

TL;DR - my test results show lz4-java 1.10.1 is about 10~15% slower on lz4 compression than 1.8.0, and is about ~5% slower on lz4 decompression even with migrating to suggested safeDecompressor

Was this patch authored or co-authored using generative AI tooling?

No.

[pan3793]

the test failure is caused by com.ibm.db2:jcc includes unshaded old lz4 classes

[pan3793]

cc @dbtsai @huaxingao, com.ibm.db2:jcc includes unshaded old lz4 classes, which causes sql/hive/docker-it modules test failure after this patch

java.lang.NoSuchMethodError: 'net.jpountz.lz4.LZ4BlockInputStream$Builder net.jpountz.lz4.LZ4BlockInputStream.newBuilder()'
 	at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:156)
 	...

I checked all versions available in Maven Central, all of them have the same issue.
https://mvnrepository.com/artifact/com.ibm.db2/jcc

I don't find the public contact info of the IBM DB2 JDBC driver team, not sure what's the next step, temporarily purge the dependency and disable DB2 tests? Or any better ideas?

---

Update: contacted the DB2 JDBC driver's author, new release that bundles the latest lz4-java is working in progress

---

Update: DB2 team provides a special JDBC driver 12.1.3.0_special_74723 that bundles lz4-java 1.10.1 which addressed NoSuchMethodError issue.

[github-actions]

JIRA Issue Information

=== Improvement SPARK-54571 ===
Summary: Use LZ4 safeDecompressor
Assignee: None
Status: Open
Affected: ["4.2.0"]

---

This comment was automatically generated by GitHub Actions

[pan3793]

this dep is only used for testing

[pan3793]

cc @dbtsai @mridulm @dongjoon-hyun @LuciferYang

[LuciferYang]

Can we upgrade it first in a separate pr？

[pan3793]

okay, let me upgrade it first

[pan3793]

@LuciferYang FYI, I opened SPARK-55136 (#53920) for it