Skip to content

[SPARK-46752][SQL][TESTS] Use default ORC compression in data source benchmarks #44777

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dongjoon-hyun wants to merge 1 commit into from
Closed

[SPARK-46752][SQL][TESTS] Use default ORC compression in data source benchmarks #44777

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 30 additions & 30 deletions sql/core/benchmarks/BuiltInDataSourceWriteBenchmark-jdk21-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,69 +2,69 @@
Parquet writer benchmark
================================================================================================

OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 5.15.0-1053-azure
AMD EPYC 7763 64-Core Processor
Parquet(PARQUET_1_0) writer benchmark: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Output Single Int Column 1641 1686 63 9.6 104.4 1.0X
Output Single Double Column 1991 2022 43 7.9 126.6 0.8X
Output Int and String Column 4249 4305 79 3.7 270.1 0.4X
Output Partitions 3165 3189 33 5.0 201.3 0.5X
Output Buckets 4429 4447 25 3.6 281.6 0.4X
Output Single Int Column 1677 1729 73 9.4 106.6 1.0X
Output Single Double Column 1879 1895 22 8.4 119.4 0.9X
Output Int and String Column 4135 4144 13 3.8 262.9 0.4X
Output Partitions 3027 3042 21 5.2 192.4 0.6X
Output Buckets 4258 4265 11 3.7 270.7 0.4X

OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 5.15.0-1053-azure
AMD EPYC 7763 64-Core Processor
Parquet(PARQUET_2_0) writer benchmark: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Output Single Int Column 1666 1688 32 9.4 105.9 1.0X
Output Single Double Column 1611 1627 24 9.8 102.4 1.0X
Output Int and String Column 4817 4829 17 3.3 306.2 0.3X
Output Partitions 3210 3237 38 4.9 204.1 0.5X
Output Buckets 4375 4384 13 3.6 278.2 0.4X
Output Single Int Column 1770 1797 38 8.9 112.5 1.0X
Output Single Double Column 1673 1703 43 9.4 106.3 1.1X
Output Int and String Column 4330 4337 10 3.6 275.3 0.4X
Output Partitions 3129 3133 6 5.0 198.9 0.6X
Output Buckets 4160 4192 46 3.8 264.5 0.4X


================================================================================================
ORC writer benchmark
================================================================================================

OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 5.15.0-1053-azure
AMD EPYC 7763 64-Core Processor
ORC writer benchmark: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Output Single Int Column 915 945 28 17.2 58.2 1.0X
Output Single Double Column 1432 1445 18 11.0 91.0 0.6X
Output Int and String Column 3749 3771 31 4.2 238.3 0.2X
Output Partitions 2458 2472 19 6.4 156.3 0.4X
Output Buckets 3339 3358 27 4.7 212.3 0.3X
Output Single Int Column 1043 1057 20 15.1 66.3 1.0X
Output Single Double Column 2217 2243 37 7.1 141.0 0.5X
Output Int and String Column 4123 4151 40 3.8 262.1 0.3X
Output Partitions 2399 2399 1 6.6 152.5 0.4X
Output Buckets 3555 3589 48 4.4 226.0 0.3X


================================================================================================
JSON writer benchmark
================================================================================================

OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 5.15.0-1053-azure
AMD EPYC 7763 64-Core Processor
JSON writer benchmark: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Output Single Int Column 1395 1431 51 11.3 88.7 1.0X
Output Single Double Column 2105 2128 33 7.5 133.8 0.7X
Output Int and String Column 3762 3773 16 4.2 239.2 0.4X
Output Partitions 2937 2957 28 5.4 186.7 0.5X
Output Buckets 3788 3822 48 4.2 240.9 0.4X
Output Single Int Column 1444 1452 11 10.9 91.8 1.0X
Output Single Double Column 2082 2090 11 7.6 132.4 0.7X
Output Int and String Column 3729 3779 71 4.2 237.1 0.4X
Output Partitions 2781 2804 32 5.7 176.8 0.5X
Output Buckets 3664 3901 335 4.3 233.0 0.4X


================================================================================================
CSV writer benchmark
================================================================================================

OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure
OpenJDK 64-Bit Server VM 21.0.2+13-LTS on Linux 5.15.0-1053-azure
AMD EPYC 7763 64-Core Processor
CSV writer benchmark: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Output Single Int Column 3511 3524 19 4.5 223.2 1.0X
Output Single Double Column 4243 4250 9 3.7 269.8 0.8X
Output Int and String Column 6532 6557 37 2.4 415.3 0.5X
Output Partitions 5369 5374 6 2.9 341.4 0.7X
Output Buckets 6524 6531 10 2.4 414.8 0.5X
Output Single Int Column 3771 3793 31 4.2 239.7 1.0X
Output Single Double Column 4404 4419 21 3.6 280.0 0.9X
Output Int and String Column 6020 6031 16 2.6 382.7 0.6X
Output Partitions 5232 5260 38 3.0 332.7 0.7X
Output Buckets 6560 6576 23 2.4 417.1 0.6X


60 changes: 30 additions & 30 deletions sql/core/benchmarks/BuiltInDataSourceWriteBenchmark-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,69 +2,69 @@
Parquet writer benchmark
================================================================================================

OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1053-azure
OpenJDK 64-Bit Server VM 17.0.10+7-LTS on Linux 5.15.0-1053-azure
AMD EPYC 7763 64-Core Processor
Parquet(PARQUET_1_0) writer benchmark: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Output Single Int Column 1673 1702 40 9.4 106.4 1.0X
Output Single Double Column 1728 1729 1 9.1 109.9 1.0X
Output Int and String Column 4300 4367 96 3.7 273.4 0.4X
Output Partitions 3065 3156 129 5.1 194.9 0.5X
Output Buckets 4114 4115 2 3.8 261.6 0.4X
Output Single Int Column 1663 1688 35 9.5 105.7 1.0X
Output Single Double Column 1748 1797 69 9.0 111.2 1.0X
Output Int and String Column 4313 4371 82 3.6 274.2 0.4X
Output Partitions 3069 3124 77 5.1 195.1 0.5X
Output Buckets 4149 4179 42 3.8 263.8 0.4X

OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1053-azure
OpenJDK 64-Bit Server VM 17.0.10+7-LTS on Linux 5.15.0-1053-azure
AMD EPYC 7763 64-Core Processor
Parquet(PARQUET_2_0) writer benchmark: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Output Single Int Column 1650 1661 15 9.5 104.9 1.0X
Output Single Double Column 1660 1670 14 9.5 105.6 1.0X
Output Int and String Column 4645 4655 14 3.4 295.3 0.4X
Output Partitions 3083 3086 5 5.1 196.0 0.5X
Output Buckets 3937 3975 53 4.0 250.3 0.4X
Output Single Int Column 1618 1619 1 9.7 102.9 1.0X
Output Single Double Column 1639 1646 10 9.6 104.2 1.0X
Output Int and String Column 4493 4493 1 3.5 285.7 0.4X
Output Partitions 3086 3098 16 5.1 196.2 0.5X
Output Buckets 4021 4054 47 3.9 255.7 0.4X


================================================================================================
ORC writer benchmark
================================================================================================

OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1053-azure
OpenJDK 64-Bit Server VM 17.0.10+7-LTS on Linux 5.15.0-1053-azure
AMD EPYC 7763 64-Core Processor
ORC writer benchmark: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Output Single Int Column 918 929 10 17.1 58.4 1.0X
Output Single Double Column 1381 1382 2 11.4 87.8 0.7X
Output Int and String Column 3952 4028 108 4.0 251.2 0.2X
Output Partitions 2315 2319 5 6.8 147.2 0.4X
Output Buckets 2904 2915 15 5.4 184.6 0.3X
Output Single Int Column 870 873 4 18.1 55.3 1.0X
Output Single Double Column 2075 2104 40 7.6 132.0 0.4X
Output Int and String Column 4568 4576 11 3.4 290.4 0.2X
Output Partitions 2489 2492 4 6.3 158.3 0.3X
Output Buckets 3555 3566 16 4.4 226.0 0.2X


================================================================================================
JSON writer benchmark
================================================================================================

OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1053-azure
OpenJDK 64-Bit Server VM 17.0.10+7-LTS on Linux 5.15.0-1053-azure
AMD EPYC 7763 64-Core Processor
JSON writer benchmark: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Output Single Int Column 1350 1358 11 11.6 85.8 1.0X
Output Single Double Column 2017 2045 39 7.8 128.2 0.7X
Output Int and String Column 3554 3593 56 4.4 225.9 0.4X
Output Partitions 2809 2816 10 5.6 178.6 0.5X
Output Buckets 3656 3667 15 4.3 232.4 0.4X
Output Single Int Column 1348 1354 8 11.7 85.7 1.0X
Output Single Double Column 1958 1970 18 8.0 124.5 0.7X
Output Int and String Column 4112 4116 5 3.8 261.5 0.3X
Output Partitions 2983 3017 49 5.3 189.6 0.5X
Output Buckets 3810 3821 15 4.1 242.2 0.4X


================================================================================================
CSV writer benchmark
================================================================================================

OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1053-azure
OpenJDK 64-Bit Server VM 17.0.10+7-LTS on Linux 5.15.0-1053-azure
AMD EPYC 7763 64-Core Processor
CSV writer benchmark: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Output Single Int Column 3151 3153 3 5.0 200.3 1.0X
Output Single Double Column 3469 3470 1 4.5 220.6 0.9X
Output Int and String Column 5851 5860 13 2.7 372.0 0.5X
Output Partitions 4830 4842 16 3.3 307.1 0.7X
Output Buckets 6158 6181 32 2.6 391.5 0.5X
Output Single Int Column 2911 2970 85 5.4 185.0 1.0X
Output Single Double Column 3668 3676 12 4.3 233.2 0.8X
Output Int and String Column 5563 5564 1 2.8 353.7 0.5X
Output Partitions 4608 4636 39 3.4 293.0 0.6X
Output Buckets 5745 5771 38 2.7 365.2 0.5X


Loading