Skip to content

Commit 5045380

Browse files
roczeiroczei
committed
[SPARK-48867][BUILD] Upgrade okhttp to 4.12.0, okio to 3.9.0 and esdk-obs-java to 3.24.3
### What changes were proposed in this pull request? This PR aims to upgrade `okhttp` to 4.12.0, `okio` to 3.9.0 and `esdk-obs-java` to 3.24.3. ### Why are the changes needed? okhttp depends on okio which has to be upgraded as well. The new okhttp version fixes the following vulnerabilities: 1) CVE-2023-0833 - A flaw was found in Red Hat's AMQ-Streams, which ships a version of the OKHttp component with an information disclosure flaw via an exception triggered by a header containing an illegal value. This issue could allow an authenticated attacker to access information outside of their regular permissions. CVSSv3 Score:- 5.5(Medium) https://nvd.nist.gov/vuln/detail/CVE-2023-0833 2) CVE-2021-0341 - In verifyHostName of OkHostnameVerifier.java, there is a possible way to accept a certificate for the wrong domain due to improperly used crypto. This could lead to remote information disclosure with no additional execution privileges needed. User interaction is not needed for exploitation. CVSSv3 Score:- 7.5(High) https://nvd.nist.gov/vuln/detail/CVE-2021-0341 square/okhttp#6724 There are two places in the Spark repository where the okhttp dependency comes in as transitive dependency: 1) [INFO] +- org.apache.hadoop:hadoop-cloud-storage:jar:3.4.0:compile [INFO] | +- org.apache.hadoop:hadoop-annotations:jar:3.4.0:compile [INFO] | +- org.apache.hadoop:hadoop-aliyun:jar:3.4.0:compile [INFO] | | +- com.aliyun.oss:aliyun-sdk-oss:jar:3.13.2:compile [INFO] | | | +- org.jdom:jdom2:jar:2.0.6:compile [INFO] | | | +- com.aliyun:aliyun-java-sdk-core:jar:4.5.10:compile [INFO] | | | | +- org.ini4j:ini4j:jar:0.5.4:compile [INFO] | | | | +- io.opentracing:opentracing-api:jar:0.33.0:compile [INFO] | | | | \- io.opentracing:opentracing-util:jar:0.33.0:compile [INFO] | | | | \- io.opentracing:opentracing-noop:jar:0.33.0:compile [INFO] | | | +- com.aliyun:aliyun-java-sdk-ram:jar:3.1.0:compile [INFO] | | | \- com.aliyun:aliyun-java-sdk-kms:jar:2.11.0:compile [INFO] | | \- org.codehaus.jettison:jettison:jar:1.5.4:compile [INFO] | +- org.apache.hadoop:hadoop-azure-datalake:jar:3.4.0:compile [INFO] | | \- com.microsoft.azure:azure-data-lake-store-sdk:jar:2.3.9:compile [INFO] | \- org.apache.hadoop:hadoop-huaweicloud:jar:3.4.0:compile [INFO] | \- com.huaweicloud:esdk-obs-java:jar:3.20.4.2:compile [INFO] | +- com.jamesmurty.utils:java-xmlbuilder:jar:1.2:compile [INFO] | +- com.squareup.okhttp3:okhttp:jar:3.14.2:compile [INFO] | \- com.squareup.okio:okio:jar:1.17.6:compile The Hadoop team has attempted to remove okhttp from their codebase: remove okhttp usage: https://issues.apache.org/jira/browse/HADOOP-18890 Unfortunately the hadoop-huaweicloud dependency is still there which pulls in the vulnerable okhttp 3.x version. https://github.com/apache/hadoop/blob/trunk/hadoop-cloud-storage-project/hadoop-cloud-storage/pom.xml#L137C19-L137C37 Proposed solution for this: com.huaweicloud:esdk-obs-java:jar:3.20.4.2 is vulnerable due to okhttp 3.x (CVE-2023-0833, CVE-2021-0341), it has to be upgraded to 3.24.3 which depends on okhttp 4.12.0 2) [INFO] +- org.apache.spark:spark-kubernetes_2.13:jar:4.0.0-SNAPSHOT:compile [INFO] | +- io.fabric8:kubernetes-httpclient-okhttp:jar:6.13.3:compile [INFO] | | +- io.fabric8:kubernetes-client-api:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-core:jar:6.13.3:compile [INFO] | | | | \- io.fabric8:kubernetes-model-common:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-gatewayapi:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-resource:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-rbac:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-admissionregistration:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-apps:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-autoscaling:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-apiextensions:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-batch:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-certificates:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-coordination:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-discovery:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-events:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-extensions:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-flowcontrol:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-networking:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-metrics:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-policy:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-scheduling:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-storageclass:jar:6.13.3:compile [INFO] | | | +- io.fabric8:kubernetes-model-node:jar:6.13.3:compile [INFO] | | | \- org.snakeyaml:snakeyaml-engine:jar:2.7:compile [INFO] | | +- com.squareup.okhttp3:okhttp:jar:3.12.12:compile [INFO] | | | \- com.squareup.okio:okio:jar:1.17.6:compile [INFO] | | \- com.squareup.okhttp3:logging-interceptor:jar:3.12.12:compile kubernet-client maintainers have decided to update okhttp from 3.x to 4.x in their upcoming version 7: fabric8io/kubernetes-client#5778 My proposed solution based on the above finding: Exclude the 3.x version and switch to use okhttp 4.x. Source: https://github.com/fabric8io/kubernetes-client/blob/main/doc/KubernetesClientWithIPv6Clusters.md It is binary backwards compatible with okhttp 3.x. More details are here: https://square.github.io/okhttp/upgrading_to_okhttp_4/ ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No.
1 parent d317002 commit 5045380

File tree

5 files changed

+144
-6
lines changed

5 files changed

+144
-6
lines changed

‎dev/deps/spark-deps-hadoop-3-hive-2.3‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ derbyshared/10.16.1.1//derbyshared-10.16.1.1.jar
6464
derbytools/10.16.1.1//derbytools-10.16.1.1.jar
6565
dropwizard-metrics-hadoop-metrics2-reporter/0.1.2//dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
6666
error_prone_annotations/2.26.1//error_prone_annotations-2.26.1.jar
67-
esdk-obs-java/3.20.4.2//esdk-obs-java-3.20.4.2.jar
67+
esdk-obs-java/3.24.3//esdk-obs-java-3.24.3.jar
6868
failureaccess/1.0.2//failureaccess-1.0.2.jar
6969
flatbuffers-java/24.3.25//flatbuffers-java-24.3.25.jar
7070
gcs-connector/hadoop3-2.2.25/shaded/gcs-connector-hadoop3-2.2.25-shaded.jar
@@ -123,7 +123,6 @@ jakarta.ws.rs-api/3.0.0//jakarta.ws.rs-api-3.0.0.jar
123123
jakarta.xml.bind-api/2.3.2//jakarta.xml.bind-api-2.3.2.jar
124124
janino/3.1.9//janino-3.1.9.jar
125125
java-diff-utils/4.12//java-diff-utils-4.12.jar
126-
java-xmlbuilder/1.2//java-xmlbuilder-1.2.jar
127126
javassist/3.30.2-GA//javassist-3.30.2-GA.jar
128127
javax.jdo/3.2.0-m3//javax.jdo-3.2.0-m3.jar
129128
javax.servlet-api/4.0.1//javax.servlet-api-4.0.1.jar
@@ -158,6 +157,7 @@ json4s-scalap_2.13/4.0.7//json4s-scalap_2.13-4.0.7.jar
158157
jsr305/3.0.0//jsr305-3.0.0.jar
159158
jta/1.1//jta-1.1.jar
160159
jul-to-slf4j/2.0.16//jul-to-slf4j-2.0.16.jar
160+
kotlin-stdlib/2.0.10//kotlin-stdlib-2.0.10.jar
161161
kryo-shaded/4.0.2//kryo-shaded-4.0.2.jar
162162
kubernetes-client-api/6.13.4//kubernetes-client-api-6.13.4.jar
163163
kubernetes-client/6.13.4//kubernetes-client-6.13.4.jar
@@ -194,7 +194,7 @@ log4j-api/2.24.1//log4j-api-2.24.1.jar
194194
log4j-core/2.24.1//log4j-core-2.24.1.jar
195195
log4j-layout-template-json/2.24.1//log4j-layout-template-json-2.24.1.jar
196196
log4j-slf4j2-impl/2.24.1//log4j-slf4j2-impl-2.24.1.jar
197-
logging-interceptor/3.12.12//logging-interceptor-3.12.12.jar
197+
logging-interceptor/4.12.0//logging-interceptor-4.12.0.jar
198198
lz4-java/1.8.0//lz4-java-1.8.0.jar
199199
metrics-core/4.2.28//metrics-core-4.2.28.jar
200200
metrics-graphite/4.2.28//metrics-graphite-4.2.28.jar
@@ -228,8 +228,9 @@ netty-transport-native-kqueue/4.1.114.Final/osx-x86_64/netty-transport-native-kq
228228
netty-transport-native-unix-common/4.1.114.Final//netty-transport-native-unix-common-4.1.114.Final.jar
229229
netty-transport/4.1.114.Final//netty-transport-4.1.114.Final.jar
230230
objenesis/3.3//objenesis-3.3.jar
231-
okhttp/3.12.12//okhttp-3.12.12.jar
232-
okio/1.17.6//okio-1.17.6.jar
231+
okhttp/4.12.0//okhttp-4.12.0.jar
232+
okio-jvm/3.9.0//okio-jvm-3.9.0.jar
233+
okio/3.9.0//okio-3.9.0.jar
233234
opencsv/2.3//opencsv-2.3.jar
234235
opentracing-api/0.33.0//opentracing-api-0.33.0.jar
235236
opentracing-noop/0.33.0//opentracing-noop-0.33.0.jar

‎hadoop-cloud/pom.xml‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -171,6 +171,41 @@
171171
<groupId>org.apache.hadoop</groupId>
172172
<artifactId>hadoop-cos</artifactId>
173173
</exclusion>
174+
<!--
175+
HADOOP-19224 / SPARK-48867: com.huaweicloud:esdk-obs-java:jar:3.20.4.2 is
176+
vulnerable due to okhttp 3.x (CVE-2023-0833, CVE-2021-0341),
177+
it has to be upgraded to 3.24.3 which depends on okhttp 4.12.0
178+
-->
179+
<exclusion>
180+
<groupId>com.huaweicloud</groupId>
181+
<artifactId>esdk-obs-java</artifactId>
182+
</exclusion>
183+
</exclusions>
184+
</dependency>
185+
<dependency>
186+
<groupId>com.huaweicloud</groupId>
187+
<artifactId>esdk-obs-java</artifactId>
188+
<version>${esdk.obs.java.version}</version>
189+
<exclusions>
190+
<exclusion>
191+
<groupId>org.jetbrains.kotlin</groupId>
192+
<artifactId>kotlin-stdlib-jdk8</artifactId>
193+
</exclusion>
194+
<exclusion>
195+
<groupId>org.jetbrains.kotlin</groupId>
196+
<artifactId>kotlin-stdlib</artifactId>
197+
</exclusion>
198+
</exclusions>
199+
</dependency>
200+
<dependency>
201+
<groupId>org.jetbrains.kotlin</groupId>
202+
<artifactId>kotlin-stdlib</artifactId>
203+
<version>${kotlin-stdlib.version}</version>
204+
<exclusions>
205+
<exclusion>
206+
<groupId>org.jetbrains</groupId>
207+
<artifactId>annotations</artifactId>
208+
</exclusion>
174209
</exclusions>
175210
</dependency>
176211
<!--

‎pom.xml‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -161,6 +161,12 @@
161161
<aws.java.sdk.v2.version>2.24.6</aws.java.sdk.v2.version>
162162
<!-- the producer is used in tests -->
163163
<aws.kinesis.producer.version>0.12.8</aws.kinesis.producer.version>
164+
<!--
165+
HADOOP-19224 / SPARK-48867: com.huaweicloud:esdk-obs-java:jar:3.20.4.2 is
166+
vulnerable due to okhttp 3.x (CVE-2023-0833, CVE-2021-0341),
167+
it has to be upgraded to 3.24.3 which depends on okhttp 4.12.0
168+
-->
169+
<esdk.obs.java.version>3.24.3</esdk.obs.java.version>
164170
<!-- Do not use 3.0.0: https://github.com/GoogleCloudDataproc/hadoop-connectors/issues/1114 -->
165171
<gcs-connector.version>hadoop3-2.2.25</gcs-connector.version>
166172
<!-- org.apache.httpcomponents/httpclient-->
@@ -237,7 +243,9 @@
237243
<!-- org.fusesource.leveldbjni will be used except on arm64 platform. -->
238244
<leveldbjni.group>org.fusesource.leveldbjni</leveldbjni.group>
239245
<kubernetes-client.version>6.13.4</kubernetes-client.version>
240-
<okio.version>1.17.6</okio.version>
246+
<okio.version>3.9.0</okio.version>
247+
<okhttp.version>4.12.0</okhttp.version>
248+
<kotlin-stdlib.version>2.0.10</kotlin-stdlib.version>
241249

242250
<test.java.home>${java.home}</test.java.home>
243251

‎resource-managers/kubernetes/core/pom.xml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,6 +109,53 @@
109109
<groupId>io.fabric8</groupId>
110110
<artifactId>kubernetes-httpclient-okhttp</artifactId>
111111
<version>${kubernetes-client.version}</version>
112+
<exclusions>
113+
<exclusion>
114+
<groupId>com.squareup.okhttp3</groupId>
115+
<artifactId>okhttp</artifactId>
116+
</exclusion>
117+
<exclusion>
118+
<groupId>com.squareup.okhttp3</groupId>
119+
<artifactId>logging-interceptor</artifactId>
120+
</exclusion>
121+
</exclusions>
122+
</dependency>
123+
<dependency>
124+
<groupId>com.squareup.okhttp3</groupId>
125+
<artifactId>okhttp</artifactId>
126+
<version>${okhttp.version}</version>
127+
<exclusions>
128+
<exclusion>
129+
<groupId>org.jetbrains.kotlin</groupId>
130+
<artifactId>kotlin-stdlib-jdk8</artifactId>
131+
</exclusion>
132+
<exclusion>
133+
<groupId>org.jetbrains.kotlin</groupId>
134+
<artifactId>kotlin-stdlib</artifactId>
135+
</exclusion>
136+
</exclusions>
137+
</dependency>
138+
<dependency>
139+
<groupId>com.squareup.okhttp3</groupId>
140+
<artifactId>logging-interceptor</artifactId>
141+
<version>${okhttp.version}</version>
142+
<exclusions>
143+
<exclusion>
144+
<groupId>org.jetbrains.kotlin</groupId>
145+
<artifactId>kotlin-stdlib-jdk8</artifactId>
146+
</exclusion>
147+
</exclusions>
148+
</dependency>
149+
<dependency>
150+
<groupId>org.jetbrains.kotlin</groupId>
151+
<artifactId>kotlin-stdlib</artifactId>
152+
<version>${kotlin-stdlib.version}</version>
153+
<exclusions>
154+
<exclusion>
155+
<groupId>org.jetbrains</groupId>
156+
<artifactId>annotations</artifactId>
157+
</exclusion>
158+
</exclusions>
112159
</dependency>
113160
<dependency>
114161
<groupId>io.fabric8</groupId>

‎resource-managers/kubernetes/integration-tests/pom.xml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,53 @@
6767
<groupId>io.fabric8</groupId>
6868
<artifactId>kubernetes-client</artifactId>
6969
<version>${kubernetes-client.version}</version>
70+
<exclusions>
71+
<exclusion>
72+
<groupId>com.squareup.okhttp3</groupId>
73+
<artifactId>okhttp</artifactId>
74+
</exclusion>
75+
<exclusion>
76+
<groupId>com.squareup.okhttp3</groupId>
77+
<artifactId>logging-interceptor</artifactId>
78+
</exclusion>
79+
</exclusions>
80+
</dependency>
81+
<dependency>
82+
<groupId>com.squareup.okhttp3</groupId>
83+
<artifactId>okhttp</artifactId>
84+
<version>${okhttp.version}</version>
85+
<exclusions>
86+
<exclusion>
87+
<groupId>org.jetbrains.kotlin</groupId>
88+
<artifactId>kotlin-stdlib-jdk8</artifactId>
89+
</exclusion>
90+
<exclusion>
91+
<groupId>org.jetbrains.kotlin</groupId>
92+
<artifactId>kotlin-stdlib</artifactId>
93+
</exclusion>
94+
</exclusions>
95+
</dependency>
96+
<dependency>
97+
<groupId>com.squareup.okhttp3</groupId>
98+
<artifactId>logging-interceptor</artifactId>
99+
<version>${okhttp.version}</version>
100+
<exclusions>
101+
<exclusion>
102+
<groupId>org.jetbrains.kotlin</groupId>
103+
<artifactId>kotlin-stdlib-jdk8</artifactId>
104+
</exclusion>
105+
</exclusions>
106+
</dependency>
107+
<dependency>
108+
<groupId>org.jetbrains.kotlin</groupId>
109+
<artifactId>kotlin-stdlib</artifactId>
110+
<version>${kotlin-stdlib.version}</version>
111+
<exclusions>
112+
<exclusion>
113+
<groupId>org.jetbrains</groupId>
114+
<artifactId>annotations</artifactId>
115+
</exclusion>
116+
</exclusions>
70117
</dependency>
71118
<dependency>
72119
<groupId>org.apache.spark</groupId>

0 commit comments

Comments
 (0)