Skip to content

Commit 78addeb

Browse files
a0x8oa0x8o
committed
[SPARK-51049][CORE] Increase S3A Vector IO threshold for range merge
### What changes were proposed in this pull request? This PR aims to increase S3A Vector IO threshold for range merge. ### Why are the changes needed? Apache Spark 4.0.0 supported Hadoop Vectored IO via ORC and Parquet. As a part of [HADOOP-18855 VectorIO API tuning/stabilization](https://issues.apache.org/jira/browse/HADOOP-18855), Apache Hadoop 3.4.2 will have new threshold default values. We had better follow these update in advance until Apache Hadoop 3.4.2 is released. - apache/hadoop#7281 ### Does this PR introduce _any_ user-facing change? No, Hadoop Vectored IO features are new in Apache Spark 4.0.0 . ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #49748 from dongjoon-hyun/SPARK-51049. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent 4ae557d commit 78addeb

File tree

230 files changed

+25347
-6861
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

230 files changed

+25347
-6861
lines changed

‎assembly/pom.xml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -192,7 +192,7 @@
192192
<executable>cp</executable>
193193
<arguments>
194194
<argument>-r</argument>
195-
<argument>${basedir}/../connector/connect/client/jvm/target/connect-repl</argument>
195+
<argument>${basedir}/../sql/connect/client/jvm/target/connect-repl</argument>
196196
<argument>${basedir}/target/scala-${scala.binary.version}/jars/</argument>
197197
</arguments>
198198
</configuration>
@@ -206,7 +206,7 @@
206206
<configuration>
207207
<executable>cp</executable>
208208
<arguments>
209-
<argument>${basedir}/../connector/connect/client/jvm/target/spark-connect-client-jvm_${scala.binary.version}-${project.version}.jar</argument>
209+
<argument>${basedir}/../sql/connect/client/jvm/target/spark-connect-client-jvm_${scala.binary.version}-${project.version}.jar</argument>
210210
<argument>${basedir}/target/scala-${scala.binary.version}/jars/connect-repl</argument>
211211
</arguments>
212212
</configuration>

‎connector/avro/benchmarks/AvroReadBenchmark-jdk21-results.txt‎

Lines changed: 38 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -2,140 +2,140 @@
22
SQL Single Numeric Column Scan
33
================================================================================================
44

5-
OpenJDK 64-Bit Server VM 21.0.5+11-LTS on Linux 6.8.0-1017-azure
5+
OpenJDK 64-Bit Server VM 21.0.6+7-LTS on Linux 6.8.0-1020-azure
66
AMD EPYC 7763 64-Core Processor
77
SQL Single TINYINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
88
------------------------------------------------------------------------------------------------------------------------
9-
Sum 1971 1989 26 8.0 125.3 1.0X
9+
Sum 2045 2062 24 7.7 130.0 1.0X
1010

11-
OpenJDK 64-Bit Server VM 21.0.5+11-LTS on Linux 6.8.0-1017-azure
11+
OpenJDK 64-Bit Server VM 21.0.6+7-LTS on Linux 6.8.0-1020-azure
1212
AMD EPYC 7763 64-Core Processor
1313
SQL Single SMALLINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
1414
------------------------------------------------------------------------------------------------------------------------
15-
Sum 1958 2014 80 8.0 124.5 1.0X
15+
Sum 2033 2040 11 7.7 129.2 1.0X
1616

17-
OpenJDK 64-Bit Server VM 21.0.5+11-LTS on Linux 6.8.0-1017-azure
17+
OpenJDK 64-Bit Server VM 21.0.6+7-LTS on Linux 6.8.0-1020-azure
1818
AMD EPYC 7763 64-Core Processor
1919
SQL Single INT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
2020
------------------------------------------------------------------------------------------------------------------------
21-
Sum 1956 1987 44 8.0 124.3 1.0X
21+
Sum 1959 1965 8 8.0 124.6 1.0X
2222

23-
OpenJDK 64-Bit Server VM 21.0.5+11-LTS on Linux 6.8.0-1017-azure
23+
OpenJDK 64-Bit Server VM 21.0.6+7-LTS on Linux 6.8.0-1020-azure
2424
AMD EPYC 7763 64-Core Processor
2525
SQL Single BIGINT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
2626
------------------------------------------------------------------------------------------------------------------------
27-
Sum 1953 1962 12 8.1 124.2 1.0X
27+
Sum 1983 2019 51 7.9 126.1 1.0X
2828

29-
OpenJDK 64-Bit Server VM 21.0.5+11-LTS on Linux 6.8.0-1017-azure
29+
OpenJDK 64-Bit Server VM 21.0.6+7-LTS on Linux 6.8.0-1020-azure
3030
AMD EPYC 7763 64-Core Processor
3131
SQL Single FLOAT Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
3232
------------------------------------------------------------------------------------------------------------------------
33-
Sum 1948 1950 3 8.1 123.9 1.0X
33+
Sum 1935 1941 7 8.1 123.1 1.0X
3434

35-
OpenJDK 64-Bit Server VM 21.0.5+11-LTS on Linux 6.8.0-1017-azure
35+
OpenJDK 64-Bit Server VM 21.0.6+7-LTS on Linux 6.8.0-1020-azure
3636
AMD EPYC 7763 64-Core Processor
3737
SQL Single DOUBLE Column Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
3838
------------------------------------------------------------------------------------------------------------------------
39-
Sum 1933 1938 6 8.1 122.9 1.0X
39+
Sum 1960 1976 23 8.0 124.6 1.0X
4040

4141

4242
================================================================================================
4343
Int and String Scan
4444
================================================================================================
4545

46-
OpenJDK 64-Bit Server VM 21.0.5+11-LTS on Linux 6.8.0-1017-azure
46+
OpenJDK 64-Bit Server VM 21.0.6+7-LTS on Linux 6.8.0-1020-azure
4747
AMD EPYC 7763 64-Core Processor
4848
Int and String Scan: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
4949
------------------------------------------------------------------------------------------------------------------------
50-
Sum of columns 3570 3574 6 2.9 340.4 1.0X
50+
Sum of columns 3561 3573 16 2.9 339.6 1.0X
5151

5252

5353
================================================================================================
5454
Partitioned Table Scan
5555
================================================================================================
5656

57-
OpenJDK 64-Bit Server VM 21.0.5+11-LTS on Linux 6.8.0-1017-azure
57+
OpenJDK 64-Bit Server VM 21.0.6+7-LTS on Linux 6.8.0-1020-azure
5858
AMD EPYC 7763 64-Core Processor
5959
Partitioned Table: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
6060
------------------------------------------------------------------------------------------------------------------------
61-
Data column 2062 2083 30 7.6 131.1 1.0X
62-
Partition column 1869 1873 5 8.4 118.9 1.1X
63-
Both columns 2057 2093 51 7.6 130.8 1.0X
61+
Data column 2138 2158 28 7.4 135.9 1.0X
62+
Partition column 1817 1828 15 8.7 115.5 1.2X
63+
Both columns 2066 2092 36 7.6 131.4 1.0X
6464

6565

6666
================================================================================================
6767
Repeated String Scan
6868
================================================================================================
6969

70-
OpenJDK 64-Bit Server VM 21.0.5+11-LTS on Linux 6.8.0-1017-azure
70+
OpenJDK 64-Bit Server VM 21.0.6+7-LTS on Linux 6.8.0-1020-azure
7171
AMD EPYC 7763 64-Core Processor
7272
Repeated String: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
7373
------------------------------------------------------------------------------------------------------------------------
74-
Sum of string length 2010 2012 3 5.2 191.7 1.0X
74+
Sum of string length 2183 2212 41 4.8 208.2 1.0X
7575

7676

7777
================================================================================================
7878
String with Nulls Scan
7979
================================================================================================
8080

81-
OpenJDK 64-Bit Server VM 21.0.5+11-LTS on Linux 6.8.0-1017-azure
81+
OpenJDK 64-Bit Server VM 21.0.6+7-LTS on Linux 6.8.0-1020-azure
8282
AMD EPYC 7763 64-Core Processor
8383
String with Nulls Scan (0.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
8484
------------------------------------------------------------------------------------------------------------------------
85-
Sum of string length 3082 3094 16 3.4 293.9 1.0X
85+
Sum of string length 3278 3296 26 3.2 312.6 1.0X
8686

87-
OpenJDK 64-Bit Server VM 21.0.5+11-LTS on Linux 6.8.0-1017-azure
87+
OpenJDK 64-Bit Server VM 21.0.6+7-LTS on Linux 6.8.0-1020-azure
8888
AMD EPYC 7763 64-Core Processor
8989
String with Nulls Scan (50.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
9090
------------------------------------------------------------------------------------------------------------------------
91-
Sum of string length 2220 2245 36 4.7 211.7 1.0X
91+
Sum of string length 2148 2152 5 4.9 204.9 1.0X
9292

93-
OpenJDK 64-Bit Server VM 21.0.5+11-LTS on Linux 6.8.0-1017-azure
93+
OpenJDK 64-Bit Server VM 21.0.6+7-LTS on Linux 6.8.0-1020-azure
9494
AMD EPYC 7763 64-Core Processor
9595
String with Nulls Scan (95.0%): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
9696
------------------------------------------------------------------------------------------------------------------------
97-
Sum of string length 1152 1159 10 9.1 109.9 1.0X
97+
Sum of string length 1128 1129 1 9.3 107.6 1.0X
9898

9999

100100
================================================================================================
101101
Select All From Wide Columns
102102
================================================================================================
103103

104-
OpenJDK 64-Bit Server VM 21.0.5+11-LTS on Linux 6.8.0-1017-azure
104+
OpenJDK 64-Bit Server VM 21.0.6+7-LTS on Linux 6.8.0-1020-azure
105105
AMD EPYC 7763 64-Core Processor
106106
Wide Column Scan from 1000 columns: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
107107
------------------------------------------------------------------------------------------------------------------------
108-
Select of all columns 20941 20946 6 0.0 41882.8 1.0X
108+
Select of all columns 20327 20405 111 0.0 40653.8 1.0X
109109

110110

111111
================================================================================================
112112
Single Column Scan From Wide Columns
113113
================================================================================================
114114

115-
OpenJDK 64-Bit Server VM 21.0.5+11-LTS on Linux 6.8.0-1017-azure
115+
OpenJDK 64-Bit Server VM 21.0.6+7-LTS on Linux 6.8.0-1020-azure
116116
AMD EPYC 7763 64-Core Processor
117117
Single Column Scan from 100 columns: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
118118
------------------------------------------------------------------------------------------------------------------------
119-
Sum of single column 3425 3440 21 0.3 3266.1 1.0X
119+
Sum of single column 3365 3372 9 0.3 3209.4 1.0X
120120

121-
OpenJDK 64-Bit Server VM 21.0.5+11-LTS on Linux 6.8.0-1017-azure
121+
OpenJDK 64-Bit Server VM 21.0.6+7-LTS on Linux 6.8.0-1020-azure
122122
AMD EPYC 7763 64-Core Processor
123123
Single Column Scan from 200 columns: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
124124
------------------------------------------------------------------------------------------------------------------------
125-
Sum of single column 6740 6770 43 0.2 6427.4 1.0X
125+
Sum of single column 6611 6633 32 0.2 6304.7 1.0X
126126

127-
OpenJDK 64-Bit Server VM 21.0.5+11-LTS on Linux 6.8.0-1017-azure
127+
OpenJDK 64-Bit Server VM 21.0.6+7-LTS on Linux 6.8.0-1020-azure
128128
AMD EPYC 7763 64-Core Processor
129129
Single Column Scan from 300 columns: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
130130
------------------------------------------------------------------------------------------------------------------------
131-
Sum of single column 9988 10056 96 0.1 9525.4 1.0X
131+
Sum of single column 10030 10036 9 0.1 9565.3 1.0X
132132

133133

134-
OpenJDK 64-Bit Server VM 21.0.5+11-LTS on Linux 6.8.0-1017-azure
134+
OpenJDK 64-Bit Server VM 21.0.6+7-LTS on Linux 6.8.0-1020-azure
135135
AMD EPYC 7763 64-Core Processor
136136
Filters pushdown: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
137137
------------------------------------------------------------------------------------------------------------------------
138-
w/o filters 5802 5844 44 0.2 5801.7 1.0X
139-
pushdown disabled 5544 5616 97 0.2 5543.9 1.0X
140-
w/ filters 2605 2609 5 0.4 2605.4 2.2X
138+
w/o filters 5455 5470 21 0.2 5455.0 1.0X
139+
pushdown disabled 5362 5367 5 0.2 5362.0 1.0X
140+
w/ filters 2367 2384 15 0.4 2367.4 2.3X
141141

0 commit comments

Comments
 (0)