Skip to content

Commit d1404e1

Browse files
roczeiroczei
committed
[SPARK-48867][BUILD] Upgrade okhttp to 4.12.0 and okio to 3.9.0
### What changes were proposed in this pull request? This PR aims to upgrade `okhttp` to 4.12.0 and `okio` to 3.9.0 ### Why are the changes needed? okhttp depends on okio which has to be upgraded as well. The new okhttp version fixes the following vulnerabilities: 1) CVE-2023-0833 - A flaw was found in Red Hat's AMQ-Streams, which ships a version of the OKHttp component with an information disclosure flaw via an exception triggered by a header containing an illegal value. This issue could allow an authenticated attacker to access information outside of their regular permissions. CVSSv3 Score:- 5.5(Medium) https://nvd.nist.gov/vuln/detail/CVE-2023-0833 2) CVE-2021-0341 - In verifyHostName of OkHostnameVerifier.java, there is a possible way to accept a certificate for the wrong domain due to improperly used crypto. This could lead to remote information disclosure with no additional execution privileges needed. User interaction is not needed for exploitation. CVSSv3 Score:- 7.5(High) https://nvd.nist.gov/vuln/detail/CVE-2021-0341 square/okhttp#6724 There are two places in the Spark repository where the okhttp dependency comes in as transitive dependency: 1) [INFO] +- org.apache.hadoop:hadoop-cloud-storage:jar:3.4.0:compile [INFO] | +- org.apache.hadoop:hadoop-annotations:jar:3.4.0:compile [INFO] | +- org.apache.hadoop:hadoop-aliyun:jar:3.4.0:compile [INFO] | | +- com.aliyun.oss:aliyun-sdk-oss:jar:3.13.2:compile [INFO] | | | +- org.jdom:jdom2:jar:2.0.6:compile [INFO] | | | +- com.aliyun:aliyun-java-sdk-core:jar:4.5.10:compile [INFO] | | | | +- org.ini4j:ini4j:jar:0.5.4:compile [INFO] | | | | +- io.opentracing:opentracing-api:jar:0.33.0:compile [INFO] | | | | \- io.opentracing:opentracing-util:jar:0.33.0:compile [INFO] | | | | \- io.opentracing:opentracing-noop:jar:0.33.0:compile [INFO] | | | +- com.aliyun:aliyun-java-sdk-ram:jar:3.1.0:compile [INFO] | | | \- com.aliyun:aliyun-java-sdk-kms:jar:2.11.0:compile [INFO] | | \- org.codehaus.jettison:jettison:jar:1.5.4:compile [INFO] | +- org.apache.hadoop:hadoop-azure-datalake:jar:3.4.0:compile [INFO] | | \- com.microsoft.azure:azure-data-lake-store-sdk:jar:2.3.9:compile [INFO] | \- org.apache.hadoop:hadoop-huaweicloud:jar:3.4.0:compile [INFO] | \- com.huaweicloud:esdk-obs-java:jar:3.20.4.2:compile [INFO] | +- com.jamesmurty.utils:java-xmlbuilder:jar:1.2:compile [INFO] | +- com.squareup.okhttp3:okhttp:jar:3.14.2:compile [INFO] | \- com.squareup.okio:okio:jar:1.17.6:compile The Hadoop team has attempted to remove okhttp from their codebase: remove okhttp usage: https://issues.apache.org/jira/browse/HADOOP-18890 Unfortunately the hadoop-huaweicloud dependency is still there which pulls in the vulnerable okhttp 3.x version. https://github.com/apache/hadoop/blob/trunk/hadoop-cloud-storage-project/hadoop-cloud-storage/pom.xml#L137C19-L137C37 3.4.0 is the only available version here: https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-huaweicloud 2) [INFO] +- org.apache.spark:spark-kubernetes_2.13:jar:4.0.0-SNAPSHOT:compile [INFO] | +- io.fabric8:kubernetes-httpclient-okhttp:jar:6.13.3:compile [INFO] | | +- io.fabric8:kubernetes-client-api:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-core:jar:6.13.3:compile [INFO] | | | | \- io.fabric8:kubernetes-model-common:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-gatewayapi:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-resource:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-rbac:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-admissionregistration:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-apps:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-autoscaling:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-apiextensions:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-batch:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-certificates:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-coordination:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-discovery:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-events:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-extensions:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-flowcontrol:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-networking:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-metrics:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-policy:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-scheduling:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-storageclass:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-node:jar:6.13.3:compile [INFO] | | | \- org.snakeyaml:snakeyaml-engine:jar:2.7:compile [INFO] | | +- com.squareup.okhttp3:okhttp:jar:3.12.12:compile [INFO] | | | \- com.squareup.okio:okio:jar:1.17.6:compile [INFO] | | \- com.squareup.okhttp3:logging-interceptor:jar:3.12.12:compile The kubernetes-client's maintainers do not want upgrade to okhttp 4.x because it's based on Kotlin, they recommend to exclude 3.x. Related documentation: https://github.com/fabric8io/kubernetes-client/blob/main/doc/KubernetesClientWithIPv6Clusters.md My proposed solution based on the above finding: I think we should exclude the 3.x version and switch to use okhttp 4.x. It is binary backwards compatible with okhttp 3.x. More details are here: https://square.github.io/okhttp/upgrading_to_okhttp_4/ ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No.
1 parent 8566bc6 commit d1404e1

File tree

5 files changed

+171
-4
lines changed

5 files changed

+171
-4
lines changed

‎dev/deps/spark-deps-hadoop-3-hive-2.3‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -154,6 +154,9 @@ json4s-scalap_2.13/4.0.7//json4s-scalap_2.13-4.0.7.jar
154154
jsr305/3.0.0//jsr305-3.0.0.jar
155155
jta/1.1//jta-1.1.jar
156156
jul-to-slf4j/2.0.16//jul-to-slf4j-2.0.16.jar
157+
kotlin-stdlib-jdk7/2.0.10//kotlin-stdlib-jdk7-2.0.10.jar
158+
kotlin-stdlib-jdk8/2.0.10//kotlin-stdlib-jdk8-2.0.10.jar
159+
kotlin-stdlib/2.0.10//kotlin-stdlib-2.0.10.jar
157160
kryo-shaded/4.0.2//kryo-shaded-4.0.2.jar
158161
kubernetes-client-api/6.13.3//kubernetes-client-api-6.13.3.jar
159162
kubernetes-client/6.13.3//kubernetes-client-6.13.3.jar
@@ -189,7 +192,7 @@ log4j-api/2.22.1//log4j-api-2.22.1.jar
189192
log4j-core/2.22.1//log4j-core-2.22.1.jar
190193
log4j-layout-template-json/2.22.1//log4j-layout-template-json-2.22.1.jar
191194
log4j-slf4j2-impl/2.22.1//log4j-slf4j2-impl-2.22.1.jar
192-
logging-interceptor/3.12.12//logging-interceptor-3.12.12.jar
195+
logging-interceptor/4.12.0//logging-interceptor-4.12.0.jar
193196
lz4-java/1.8.0//lz4-java-1.8.0.jar
194197
metrics-core/4.2.26//metrics-core-4.2.26.jar
195198
metrics-graphite/4.2.26//metrics-graphite-4.2.26.jar
@@ -223,8 +226,9 @@ netty-transport-native-kqueue/4.1.110.Final/osx-x86_64/netty-transport-native-kq
223226
netty-transport-native-unix-common/4.1.110.Final//netty-transport-native-unix-common-4.1.110.Final.jar
224227
netty-transport/4.1.110.Final//netty-transport-4.1.110.Final.jar
225228
objenesis/3.3//objenesis-3.3.jar
226-
okhttp/3.12.12//okhttp-3.12.12.jar
227-
okio/1.17.6//okio-1.17.6.jar
229+
okhttp/4.12.0//okhttp-4.12.0.jar
230+
okio-jvm/3.9.0//okio-jvm-3.9.0.jar
231+
okio/3.9.0//okio-3.9.0.jar
228232
opencsv/2.3//opencsv-2.3.jar
229233
opentracing-api/0.33.0//opentracing-api-0.33.0.jar
230234
opentracing-noop/0.33.0//opentracing-noop-0.33.0.jar

‎hadoop-cloud/pom.xml‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -171,6 +171,51 @@
171171
<groupId>org.apache.hadoop</groupId>
172172
<artifactId>hadoop-cos</artifactId>
173173
</exclusion>
174+
<exclusion>
175+
<groupId>com.squareup.okhttp3</groupId>
176+
<artifactId>okhttp</artifactId>
177+
</exclusion>
178+
<exclusion>
179+
<groupId>com.squareup.okio</groupId>
180+
<artifactId>okio</artifactId>
181+
</exclusion>
182+
</exclusions>
183+
</dependency>
184+
<dependency>
185+
<groupId>com.squareup.okhttp3</groupId>
186+
<artifactId>okhttp</artifactId>
187+
<version>${okhttp.version}</version>
188+
<exclusions>
189+
<exclusion>
190+
<groupId>org.jetbrains.kotlin</groupId>
191+
<artifactId>kotlin-stdlib-jdk8</artifactId>
192+
</exclusion>
193+
<exclusion>
194+
<groupId>org.jetbrains.kotlin</groupId>
195+
<artifactId>kotlin-stdlib</artifactId>
196+
</exclusion>
197+
</exclusions>
198+
</dependency>
199+
<dependency>
200+
<groupId>org.jetbrains.kotlin</groupId>
201+
<artifactId>kotlin-stdlib-jdk8</artifactId>
202+
<version>${kotlin-stdlib.version}</version>
203+
<exclusions>
204+
<exclusion>
205+
<groupId>org.jetbrains</groupId>
206+
<artifactId>annotations</artifactId>
207+
</exclusion>
208+
</exclusions>
209+
</dependency>
210+
<dependency>
211+
<groupId>org.jetbrains.kotlin</groupId>
212+
<artifactId>kotlin-stdlib</artifactId>
213+
<version>${kotlin-stdlib.version}</version>
214+
<exclusions>
215+
<exclusion>
216+
<groupId>org.jetbrains</groupId>
217+
<artifactId>annotations</artifactId>
218+
</exclusion>
174219
</exclusions>
175220
</dependency>
176221
<!--

‎pom.xml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -231,7 +231,9 @@
231231
<!-- org.fusesource.leveldbjni will be used except on arm64 platform. -->
232232
<leveldbjni.group>org.fusesource.leveldbjni</leveldbjni.group>
233233
<kubernetes-client.version>6.13.3</kubernetes-client.version>
234-
<okio.version>1.17.6</okio.version>
234+
<okio.version>3.9.0</okio.version>
235+
<okhttp.version>4.12.0</okhttp.version>
236+
<kotlin-stdlib.version>2.0.10</kotlin-stdlib.version>
235237

236238
<test.java.home>${java.home}</test.java.home>
237239

‎resource-managers/kubernetes/core/pom.xml‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,64 @@
7979
<groupId>io.fabric8</groupId>
8080
<artifactId>kubernetes-httpclient-okhttp</artifactId>
8181
<version>${kubernetes-client.version}</version>
82+
<exclusions>
83+
<exclusion>
84+
<groupId>com.squareup.okhttp3</groupId>
85+
<artifactId>okhttp</artifactId>
86+
</exclusion>
87+
<exclusion>
88+
<groupId>com.squareup.okhttp3</groupId>
89+
<artifactId>logging-interceptor</artifactId>
90+
</exclusion>
91+
</exclusions>
92+
</dependency>
93+
<dependency>
94+
<groupId>com.squareup.okhttp3</groupId>
95+
<artifactId>okhttp</artifactId>
96+
<version>${okhttp.version}</version>
97+
<exclusions>
98+
<exclusion>
99+
<groupId>org.jetbrains.kotlin</groupId>
100+
<artifactId>kotlin-stdlib-jdk8</artifactId>
101+
</exclusion>
102+
<exclusion>
103+
<groupId>org.jetbrains.kotlin</groupId>
104+
<artifactId>kotlin-stdlib</artifactId>
105+
</exclusion>
106+
</exclusions>
107+
</dependency>
108+
<dependency>
109+
<groupId>com.squareup.okhttp3</groupId>
110+
<artifactId>logging-interceptor</artifactId>
111+
<version>${okhttp.version}</version>
112+
<exclusions>
113+
<exclusion>
114+
<groupId>org.jetbrains.kotlin</groupId>
115+
<artifactId>kotlin-stdlib-jdk8</artifactId>
116+
</exclusion>
117+
</exclusions>
118+
</dependency>
119+
<dependency>
120+
<groupId>org.jetbrains.kotlin</groupId>
121+
<artifactId>kotlin-stdlib-jdk8</artifactId>
122+
<version>${kotlin-stdlib.version}</version>
123+
<exclusions>
124+
<exclusion>
125+
<groupId>org.jetbrains</groupId>
126+
<artifactId>annotations</artifactId>
127+
</exclusion>
128+
</exclusions>
129+
</dependency>
130+
<dependency>
131+
<groupId>org.jetbrains.kotlin</groupId>
132+
<artifactId>kotlin-stdlib</artifactId>
133+
<version>${kotlin-stdlib.version}</version>
134+
<exclusions>
135+
<exclusion>
136+
<groupId>org.jetbrains</groupId>
137+
<artifactId>annotations</artifactId>
138+
</exclusion>
139+
</exclusions>
82140
</dependency>
83141
<dependency>
84142
<groupId>io.fabric8</groupId>

‎resource-managers/kubernetes/integration-tests/pom.xml‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,64 @@
6868
<groupId>io.fabric8</groupId>
6969
<artifactId>kubernetes-client</artifactId>
7070
<version>${kubernetes-client.version}</version>
71+
<exclusions>
72+
<exclusion>
73+
<groupId>com.squareup.okhttp3</groupId>
74+
<artifactId>okhttp</artifactId>
75+
</exclusion>
76+
<exclusion>
77+
<groupId>com.squareup.okhttp3</groupId>
78+
<artifactId>logging-interceptor</artifactId>
79+
</exclusion>
80+
</exclusions>
81+
</dependency>
82+
<dependency>
83+
<groupId>com.squareup.okhttp3</groupId>
84+
<artifactId>okhttp</artifactId>
85+
<version>${okhttp.version}</version>
86+
<exclusions>
87+
<exclusion>
88+
<groupId>org.jetbrains.kotlin</groupId>
89+
<artifactId>kotlin-stdlib-jdk8</artifactId>
90+
</exclusion>
91+
<exclusion>
92+
<groupId>org.jetbrains.kotlin</groupId>
93+
<artifactId>kotlin-stdlib</artifactId>
94+
</exclusion>
95+
</exclusions>
96+
</dependency>
97+
<dependency>
98+
<groupId>com.squareup.okhttp3</groupId>
99+
<artifactId>logging-interceptor</artifactId>
100+
<version>${okhttp.version}</version>
101+
<exclusions>
102+
<exclusion>
103+
<groupId>org.jetbrains.kotlin</groupId>
104+
<artifactId>kotlin-stdlib-jdk8</artifactId>
105+
</exclusion>
106+
</exclusions>
107+
</dependency>
108+
<dependency>
109+
<groupId>org.jetbrains.kotlin</groupId>
110+
<artifactId>kotlin-stdlib-jdk8</artifactId>
111+
<version>${kotlin-stdlib.version}</version>
112+
<exclusions>
113+
<exclusion>
114+
<groupId>org.jetbrains</groupId>
115+
<artifactId>annotations</artifactId>
116+
</exclusion>
117+
</exclusions>
118+
</dependency>
119+
<dependency>
120+
<groupId>org.jetbrains.kotlin</groupId>
121+
<artifactId>kotlin-stdlib</artifactId>
122+
<version>${kotlin-stdlib.version}</version>
123+
<exclusions>
124+
<exclusion>
125+
<groupId>org.jetbrains</groupId>
126+
<artifactId>annotations</artifactId>
127+
</exclusion>
128+
</exclusions>
71129
</dependency>
72130
<dependency>
73131
<groupId>org.apache.spark</groupId>

0 commit comments

Comments
 (0)