Skip to content

Commit a202e72

Browse files
roczeiroczei
committed
[SPARK-48867][BUILD] Upgrade okhttp to 4.12.0 and okio to 3.9.0
### What changes were proposed in this pull request? This PR aims to upgrade `okhttp` to 4.12.0 and `okio` to 3.9.0 ### Why are the changes needed? okhttp depends on okio which has to be upgraded as well. The new okhttp version fixes the following vulnerabilities: 1) CVE-2023-0833 - A flaw was found in Red Hat's AMQ-Streams, which ships a version of the OKHttp component with an information disclosure flaw via an exception triggered by a header containing an illegal value. This issue could allow an authenticated attacker to access information outside of their regular permissions. CVSSv3 Score:- 5.5(Medium) https://nvd.nist.gov/vuln/detail/CVE-2023-0833 2) CVE-2021-0341 - In verifyHostName of OkHostnameVerifier.java, there is a possible way to accept a certificate for the wrong domain due to improperly used crypto. This could lead to remote information disclosure with no additional execution privileges needed. User interaction is not needed for exploitation. CVSSv3 Score:- 7.5(High) https://nvd.nist.gov/vuln/detail/CVE-2021-0341 square/okhttp#6724 There are two places in the Spark repository where the okhttp dependency comes in as transitive dependency: 1) [INFO] +- org.apache.hadoop:hadoop-cloud-storage:jar:3.4.0:compile [INFO] | +- org.apache.hadoop:hadoop-annotations:jar:3.4.0:compile [INFO] | +- org.apache.hadoop:hadoop-aliyun:jar:3.4.0:compile [INFO] | | +- com.aliyun.oss:aliyun-sdk-oss:jar:3.13.2:compile [INFO] | | | +- org.jdom:jdom2:jar:2.0.6:compile [INFO] | | | +- com.aliyun:aliyun-java-sdk-core:jar:4.5.10:compile [INFO] | | | | +- org.ini4j:ini4j:jar:0.5.4:compile [INFO] | | | | +- io.opentracing:opentracing-api:jar:0.33.0:compile [INFO] | | | | \- io.opentracing:opentracing-util:jar:0.33.0:compile [INFO] | | | | \- io.opentracing:opentracing-noop:jar:0.33.0:compile [INFO] | | | +- com.aliyun:aliyun-java-sdk-ram:jar:3.1.0:compile [INFO] | | | \- com.aliyun:aliyun-java-sdk-kms:jar:2.11.0:compile [INFO] | | \- org.codehaus.jettison:jettison:jar:1.5.4:compile [INFO] | +- org.apache.hadoop:hadoop-azure-datalake:jar:3.4.0:compile [INFO] | | \- com.microsoft.azure:azure-data-lake-store-sdk:jar:2.3.9:compile [INFO] | \- org.apache.hadoop:hadoop-huaweicloud:jar:3.4.0:compile [INFO] | \- com.huaweicloud:esdk-obs-java:jar:3.20.4.2:compile [INFO] | +- com.jamesmurty.utils:java-xmlbuilder:jar:1.2:compile [INFO] | +- com.squareup.okhttp3:okhttp:jar:3.14.2:compile [INFO] | \- com.squareup.okio:okio:jar:1.17.6:compile The Hadoop team has attempted to remove okhttp from their codebase: remove okhttp usage: https://issues.apache.org/jira/browse/HADOOP-18890 Unfortunately the hadoop-huaweicloud dependency is still there which pulls in the vulnerable okhttp 3.x version. https://github.com/apache/hadoop/blob/trunk/hadoop-cloud-storage-project/hadoop-cloud-storage/pom.xml#L137C19-L137C37 3.4.0 is the only available version here: https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-huaweicloud 2) [INFO] +- org.apache.spark:spark-kubernetes_2.13:jar:4.0.0-SNAPSHOT:compile [INFO] | +- io.fabric8:kubernetes-httpclient-okhttp:jar:6.13.3:compile [INFO] | | +- io.fabric8:kubernetes-client-api:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-core:jar:6.13.3:compile [INFO] | | | | \- io.fabric8:kubernetes-model-common:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-gatewayapi:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-resource:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-rbac:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-admissionregistration:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-apps:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-autoscaling:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-apiextensions:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-batch:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-certificates:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-coordination:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-discovery:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-events:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-extensions:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-flowcontrol:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-networking:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-metrics:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-policy:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-scheduling:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-storageclass:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-node:jar:6.13.3:compile [INFO] | | | \- org.snakeyaml:snakeyaml-engine:jar:2.7:compile [INFO] | | +- com.squareup.okhttp3:okhttp:jar:3.12.12:compile [INFO] | | | \- com.squareup.okio:okio:jar:1.17.6:compile [INFO] | | \- com.squareup.okhttp3:logging-interceptor:jar:3.12.12:compile The kubernetes-client's maintainers do not want upgrade to okhttp 4.x because it's based on Kotlin, they recommend to exclude 3.x. Related documentation: https://github.com/fabric8io/kubernetes-client/blob/main/doc/KubernetesClientWithIPv6Clusters.md My proposed solution based on the above finding: I think we should exclude the 3.x version and switch to use okhttp 4.x. It is binary backwards compatible with okhttp 3.x. More details are here: https://square.github.io/okhttp/upgrading_to_okhttp_4/ ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No.
1 parent 8566bc6 commit a202e72

File tree

5 files changed

+136
-4
lines changed

5 files changed

+136
-4
lines changed

‎dev/deps/spark-deps-hadoop-3-hive-2.3‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -154,6 +154,7 @@ json4s-scalap_2.13/4.0.7//json4s-scalap_2.13-4.0.7.jar
154154
jsr305/3.0.0//jsr305-3.0.0.jar
155155
jta/1.1//jta-1.1.jar
156156
jul-to-slf4j/2.0.16//jul-to-slf4j-2.0.16.jar
157+
kotlin-stdlib/2.0.10//kotlin-stdlib-2.0.10.jar
157158
kryo-shaded/4.0.2//kryo-shaded-4.0.2.jar
158159
kubernetes-client-api/6.13.3//kubernetes-client-api-6.13.3.jar
159160
kubernetes-client/6.13.3//kubernetes-client-6.13.3.jar
@@ -189,7 +190,7 @@ log4j-api/2.22.1//log4j-api-2.22.1.jar
189190
log4j-core/2.22.1//log4j-core-2.22.1.jar
190191
log4j-layout-template-json/2.22.1//log4j-layout-template-json-2.22.1.jar
191192
log4j-slf4j2-impl/2.22.1//log4j-slf4j2-impl-2.22.1.jar
192-
logging-interceptor/3.12.12//logging-interceptor-3.12.12.jar
193+
logging-interceptor/4.12.0//logging-interceptor-4.12.0.jar
193194
lz4-java/1.8.0//lz4-java-1.8.0.jar
194195
metrics-core/4.2.26//metrics-core-4.2.26.jar
195196
metrics-graphite/4.2.26//metrics-graphite-4.2.26.jar
@@ -223,8 +224,9 @@ netty-transport-native-kqueue/4.1.110.Final/osx-x86_64/netty-transport-native-kq
223224
netty-transport-native-unix-common/4.1.110.Final//netty-transport-native-unix-common-4.1.110.Final.jar
224225
netty-transport/4.1.110.Final//netty-transport-4.1.110.Final.jar
225226
objenesis/3.3//objenesis-3.3.jar
226-
okhttp/3.12.12//okhttp-3.12.12.jar
227-
okio/1.17.6//okio-1.17.6.jar
227+
okhttp/4.12.0//okhttp-4.12.0.jar
228+
okio-jvm/3.9.0//okio-jvm-3.9.0.jar
229+
okio/3.9.0//okio-3.9.0.jar
228230
opencsv/2.3//opencsv-2.3.jar
229231
opentracing-api/0.33.0//opentracing-api-0.33.0.jar
230232
opentracing-noop/0.33.0//opentracing-noop-0.33.0.jar

‎hadoop-cloud/pom.xml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -171,6 +171,40 @@
171171
<groupId>org.apache.hadoop</groupId>
172172
<artifactId>hadoop-cos</artifactId>
173173
</exclusion>
174+
<exclusion>
175+
<groupId>com.squareup.okhttp3</groupId>
176+
<artifactId>okhttp</artifactId>
177+
</exclusion>
178+
<exclusion>
179+
<groupId>com.squareup.okio</groupId>
180+
<artifactId>okio</artifactId>
181+
</exclusion>
182+
</exclusions>
183+
</dependency>
184+
<dependency>
185+
<groupId>com.squareup.okhttp3</groupId>
186+
<artifactId>okhttp</artifactId>
187+
<version>${okhttp.version}</version>
188+
<exclusions>
189+
<exclusion>
190+
<groupId>org.jetbrains.kotlin</groupId>
191+
<artifactId>kotlin-stdlib-jdk8</artifactId>
192+
</exclusion>
193+
<exclusion>
194+
<groupId>org.jetbrains.kotlin</groupId>
195+
<artifactId>kotlin-stdlib</artifactId>
196+
</exclusion>
197+
</exclusions>
198+
</dependency>
199+
<dependency>
200+
<groupId>org.jetbrains.kotlin</groupId>
201+
<artifactId>kotlin-stdlib</artifactId>
202+
<version>${kotlin-stdlib.version}</version>
203+
<exclusions>
204+
<exclusion>
205+
<groupId>org.jetbrains</groupId>
206+
<artifactId>annotations</artifactId>
207+
</exclusion>
174208
</exclusions>
175209
</dependency>
176210
<!--

‎pom.xml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -231,7 +231,9 @@
231231
<!-- org.fusesource.leveldbjni will be used except on arm64 platform. -->
232232
<leveldbjni.group>org.fusesource.leveldbjni</leveldbjni.group>
233233
<kubernetes-client.version>6.13.3</kubernetes-client.version>
234-
<okio.version>1.17.6</okio.version>
234+
<okio.version>3.9.0</okio.version>
235+
<okhttp.version>4.12.0</okhttp.version>
236+
<kotlin-stdlib.version>2.0.10</kotlin-stdlib.version>
235237

236238
<test.java.home>${java.home}</test.java.home>
237239

‎resource-managers/kubernetes/core/pom.xml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,53 @@
7979
<groupId>io.fabric8</groupId>
8080
<artifactId>kubernetes-httpclient-okhttp</artifactId>
8181
<version>${kubernetes-client.version}</version>
82+
<exclusions>
83+
<exclusion>
84+
<groupId>com.squareup.okhttp3</groupId>
85+
<artifactId>okhttp</artifactId>
86+
</exclusion>
87+
<exclusion>
88+
<groupId>com.squareup.okhttp3</groupId>
89+
<artifactId>logging-interceptor</artifactId>
90+
</exclusion>
91+
</exclusions>
92+
</dependency>
93+
<dependency>
94+
<groupId>com.squareup.okhttp3</groupId>
95+
<artifactId>okhttp</artifactId>
96+
<version>${okhttp.version}</version>
97+
<exclusions>
98+
<exclusion>
99+
<groupId>org.jetbrains.kotlin</groupId>
100+
<artifactId>kotlin-stdlib-jdk8</artifactId>
101+
</exclusion>
102+
<exclusion>
103+
<groupId>org.jetbrains.kotlin</groupId>
104+
<artifactId>kotlin-stdlib</artifactId>
105+
</exclusion>
106+
</exclusions>
107+
</dependency>
108+
<dependency>
109+
<groupId>com.squareup.okhttp3</groupId>
110+
<artifactId>logging-interceptor</artifactId>
111+
<version>${okhttp.version}</version>
112+
<exclusions>
113+
<exclusion>
114+
<groupId>org.jetbrains.kotlin</groupId>
115+
<artifactId>kotlin-stdlib-jdk8</artifactId>
116+
</exclusion>
117+
</exclusions>
118+
</dependency>
119+
<dependency>
120+
<groupId>org.jetbrains.kotlin</groupId>
121+
<artifactId>kotlin-stdlib</artifactId>
122+
<version>${kotlin-stdlib.version}</version>
123+
<exclusions>
124+
<exclusion>
125+
<groupId>org.jetbrains</groupId>
126+
<artifactId>annotations</artifactId>
127+
</exclusion>
128+
</exclusions>
82129
</dependency>
83130
<dependency>
84131
<groupId>io.fabric8</groupId>

‎resource-managers/kubernetes/integration-tests/pom.xml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,53 @@
6868
<groupId>io.fabric8</groupId>
6969
<artifactId>kubernetes-client</artifactId>
7070
<version>${kubernetes-client.version}</version>
71+
<exclusions>
72+
<exclusion>
73+
<groupId>com.squareup.okhttp3</groupId>
74+
<artifactId>okhttp</artifactId>
75+
</exclusion>
76+
<exclusion>
77+
<groupId>com.squareup.okhttp3</groupId>
78+
<artifactId>logging-interceptor</artifactId>
79+
</exclusion>
80+
</exclusions>
81+
</dependency>
82+
<dependency>
83+
<groupId>com.squareup.okhttp3</groupId>
84+
<artifactId>okhttp</artifactId>
85+
<version>${okhttp.version}</version>
86+
<exclusions>
87+
<exclusion>
88+
<groupId>org.jetbrains.kotlin</groupId>
89+
<artifactId>kotlin-stdlib-jdk8</artifactId>
90+
</exclusion>
91+
<exclusion>
92+
<groupId>org.jetbrains.kotlin</groupId>
93+
<artifactId>kotlin-stdlib</artifactId>
94+
</exclusion>
95+
</exclusions>
96+
</dependency>
97+
<dependency>
98+
<groupId>com.squareup.okhttp3</groupId>
99+
<artifactId>logging-interceptor</artifactId>
100+
<version>${okhttp.version}</version>
101+
<exclusions>
102+
<exclusion>
103+
<groupId>org.jetbrains.kotlin</groupId>
104+
<artifactId>kotlin-stdlib-jdk8</artifactId>
105+
</exclusion>
106+
</exclusions>
107+
</dependency>
108+
<dependency>
109+
<groupId>org.jetbrains.kotlin</groupId>
110+
<artifactId>kotlin-stdlib</artifactId>
111+
<version>${kotlin-stdlib.version}</version>
112+
<exclusions>
113+
<exclusion>
114+
<groupId>org.jetbrains</groupId>
115+
<artifactId>annotations</artifactId>
116+
</exclusion>
117+
</exclusions>
71118
</dependency>
72119
<dependency>
73120
<groupId>org.apache.spark</groupId>

0 commit comments

Comments
 (0)