Skip to content

[SPARK-35670][BUILD] Upgrade ZSTD-JNI to 1.5.0-2 #32826

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dchristle wants to merge 4 commits into from
Closed

[SPARK-35670][BUILD] Upgrade ZSTD-JNI to 1.5.0-2 #32826

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
a44b866
[SPARK-35670][BUILD] Upgrade ZSTD-JNI to 1.5.0-2
dchristle Jun 7, 2021
2799e6a
Update pom.xml
dchristle Jun 16, 2021
016641f
benchmark
dongjoon-hyun Jun 17, 2021
4a567dd
Merge pull request #1 from dongjoon-hyun/SPARK-PR-32826
dchristle Jun 17, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 16 additions & 16 deletions core/benchmarks/ZStandardBenchmark-jdk11-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,26 +2,26 @@
Benchmark ZStandardCompressionCodec
================================================================================================

OpenJDK 64-Bit Server VM 11.0.10+9-LTS on Linux 5.4.0-1043-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.8.0-1033-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Benchmark ZStandardCompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------------
Compression 10000 times at level 1 without buffer pool 606 614 6 0.0 60645.3 1.0X
Compression 10000 times at level 2 without buffer pool 686 693 7 0.0 68594.9 0.9X
Compression 10000 times at level 3 without buffer pool 906 920 14 0.0 90642.7 0.7X
Compression 10000 times at level 1 with buffer pool 389 403 20 0.0 38901.4 1.6X
Compression 10000 times at level 2 with buffer pool 450 466 13 0.0 45032.0 1.3X
Compression 10000 times at level 3 with buffer pool 680 682 2 0.0 68004.2 0.9X
Compression 10000 times at level 1 without buffer pool 805 1103 500 0.0 80501.4 1.0X
Compression 10000 times at level 2 without buffer pool 728 744 20 0.0 72819.9 1.1X
Compression 10000 times at level 3 without buffer pool 987 995 7 0.0 98719.4 0.8X
Compression 10000 times at level 1 with buffer pool 371 377 8 0.0 37092.3 2.2X
Compression 10000 times at level 2 with buffer pool 465 473 6 0.0 46509.8 1.7X
Compression 10000 times at level 3 with buffer pool 715 738 20 0.0 71500.2 1.1X

OpenJDK 64-Bit Server VM 11.0.10+9-LTS on Linux 5.4.0-1043-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
OpenJDK 64-Bit Server VM 11.0.11+9-LTS on Linux 5.8.0-1033-azure
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Benchmark ZStandardCompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------------------------
Decompression 10000 times from level 1 without buffer pool 1209 1226 25 0.0 120862.8 1.0X
Decompression 10000 times from level 2 without buffer pool 1191 1193 3 0.0 119064.9 1.0X
Decompression 10000 times from level 3 without buffer pool 1188 1193 6 0.0 118843.3 1.0X
Decompression 10000 times from level 1 with buffer pool 998 1004 9 0.0 99754.7 1.2X
Decompression 10000 times from level 2 with buffer pool 990 1001 11 0.0 99043.8 1.2X
Decompression 10000 times from level 3 with buffer pool 983 999 20 0.0 98269.5 1.2X
Decompression 10000 times from level 1 without buffer pool 776 786 11 0.0 77649.5 1.0X
Decompression 10000 times from level 2 without buffer pool 787 792 5 0.0 78686.6 1.0X
Decompression 10000 times from level 3 without buffer pool 782 790 7 0.0 78195.4 1.0X
Decompression 10000 times from level 1 with buffer pool 529 551 21 0.0 52914.0 1.5X
Decompression 10000 times from level 2 with buffer pool 523 537 11 0.0 52266.2 1.5X
Decompression 10000 times from level 3 with buffer pool 519 527 10 0.0 51932.3 1.5X


32 changes: 16 additions & 16 deletions core/benchmarks/ZStandardBenchmark-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,26 +2,26 @@
Benchmark ZStandardCompressionCodec
================================================================================================

OpenJDK 64-Bit Server VM 1.8.0_282-b08 on Linux 5.4.0-1043-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
OpenJDK 64-Bit Server VM 1.8.0_292-b10 on Linux 5.8.0-1033-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Benchmark ZStandardCompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------------
Compression 10000 times at level 1 without buffer pool 670 681 9 0.0 67011.0 1.0X
Compression 10000 times at level 2 without buffer pool 569 571 2 0.0 56932.0 1.2X
Compression 10000 times at level 3 without buffer pool 748 751 2 0.0 74813.8 0.9X
Compression 10000 times at level 1 with buffer pool 336 337 1 0.0 33630.6 2.0X
Compression 10000 times at level 2 with buffer pool 395 397 2 0.0 39472.6 1.7X
Compression 10000 times at level 3 with buffer pool 563 567 4 0.0 56272.8 1.2X
Compression 10000 times at level 1 without buffer pool 444 606 183 0.0 44440.9 1.0X
Compression 10000 times at level 2 without buffer pool 514 527 10 0.0 51421.8 0.9X
Compression 10000 times at level 3 without buffer pool 725 729 6 0.0 72531.4 0.6X
Compression 10000 times at level 1 with buffer pool 229 235 6 0.0 22886.7 1.9X
Compression 10000 times at level 2 with buffer pool 288 303 15 0.0 28802.3 1.5X
Compression 10000 times at level 3 with buffer pool 493 521 26 0.0 49339.5 0.9X

OpenJDK 64-Bit Server VM 1.8.0_282-b08 on Linux 5.4.0-1043-azure
Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
OpenJDK 64-Bit Server VM 1.8.0_292-b10 on Linux 5.8.0-1033-azure
Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
Benchmark ZStandardCompressionCodec: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------------------------
Decompression 10000 times from level 1 without buffer pool 1029 1031 3 0.0 102887.4 1.0X
Decompression 10000 times from level 2 without buffer pool 1028 1031 4 0.0 102847.8 1.0X
Decompression 10000 times from level 3 without buffer pool 1029 1029 0 0.0 102941.0 1.0X
Decompression 10000 times from level 1 with buffer pool 798 799 0 0.0 79838.0 1.3X
Decompression 10000 times from level 2 with buffer pool 799 799 0 0.0 79852.9 1.3X
Decompression 10000 times from level 3 with buffer pool 796 798 2 0.0 79630.5 1.3X
Decompression 10000 times from level 1 without buffer pool 1188 1192 6 0.0 118770.4 1.0X
Decompression 10000 times from level 2 without buffer pool 1176 1199 33 0.0 117574.4 1.0X
Decompression 10000 times from level 3 without buffer pool 1174 1175 1 0.0 117426.0 1.0X
Decompression 10000 times from level 1 with buffer pool 1020 1046 36 0.0 102021.9 1.2X
Decompression 10000 times from level 2 with buffer pool 996 1005 14 0.0 99561.0 1.2X
Decompression 10000 times from level 3 with buffer pool 1021 1022 1 0.0 102050.9 1.2X


2 changes: 1 addition & 1 deletion dev/deps/spark-deps-hadoop-2.7-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -242,4 +242,4 @@ xz/1.8//xz-1.8.jar
zjsonpatch/0.3.0//zjsonpatch-0.3.0.jar
zookeeper-jute/3.6.2//zookeeper-jute-3.6.2.jar
zookeeper/3.6.2//zookeeper-3.6.2.jar
zstd-jni/1.4.9-1//zstd-jni-1.4.9-1.jar
zstd-jni/1.5.0-2//zstd-jni-1.5.0-2.jar
2 changes: 1 addition & 1 deletion dev/deps/spark-deps-hadoop-3.2-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -209,4 +209,4 @@ xz/1.8//xz-1.8.jar
zjsonpatch/0.3.0//zjsonpatch-0.3.0.jar
zookeeper-jute/3.6.2//zookeeper-jute-3.6.2.jar
zookeeper/3.6.2//zookeeper-3.6.2.jar
zstd-jni/1.4.9-1//zstd-jni-1.4.9-1.jar
zstd-jni/1.5.0-2//zstd-jni-1.5.0-2.jar
2 changes: 1 addition & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -702,7 +702,7 @@
<dependency>
<groupId>com.github.luben</groupId>
<artifactId>zstd-jni</artifactId>
<version>1.4.9-1</version>
<version>1.5.0-2</version>
</dependency>
<dependency>
<groupId>com.clearspring.analytics</groupId>
Expand Down