Skip to content

Commit f1f04b0

Browse files
pan3793dongjoon-hyun
pan3793
authored and
dongjoon-hyun
committed
[SPARK-49988][BUILD] Remove unused Hadoop dependency management
### What changes were proposed in this pull request? Remove unused vanilla hadoop dependency(and transitive deps) management, i.e. `hadoop-client`, `xerces:xercesImpl`, and inline deps defined in `hadoop3` because it's the only supported hadoop profile. ### Why are the changes needed? Simplify pom.xml. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass CI and verified runtime jars are not affected by running`dev/test-dependencies.sh`. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#48491 from pan3793/SPARK-49988. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent b078c0d commit f1f04b0

File tree

5 files changed

+6
-334
lines changed

5 files changed

+6
-334
lines changed

‎LICENSE-binary

-1
Original file line numberDiff line numberDiff line change
@@ -402,7 +402,6 @@ org.xerial.snappy:snappy-java
402402
org.yaml:snakeyaml
403403
oro:oro
404404
stax:stax-api
405-
xerces:xercesImpl
406405

407406
core/src/main/java/org/apache/spark/util/collection/TimSort.java
408407
core/src/main/resources/org/apache/spark/ui/static/bootstrap*

‎NOTICE-binary

-21
Original file line numberDiff line numberDiff line change
@@ -448,27 +448,6 @@ which has the following notices:
448448
* Alec Wysoker
449449
* Performance and memory usage improvement
450450

451-
The binary distribution of this product bundles binaries of
452-
Xerces2 Java Parser 2.9.1,
453-
which has the following notices:
454-
* =========================================================================
455-
== NOTICE file corresponding to section 4(d) of the Apache License, ==
456-
== Version 2.0, in this case for the Apache Xerces Java distribution. ==
457-
=========================================================================
458-
459-
Apache Xerces Java
460-
Copyright 1999-2007 The Apache Software Foundation
461-
462-
This product includes software developed at
463-
The Apache Software Foundation (http://www.apache.org/).
464-
465-
Portions of this software were originally based on the following:
466-
- software copyright (c) 1999, IBM Corporation., http://www.ibm.com.
467-
- software copyright (c) 1999, Sun Microsystems., http://www.sun.com.
468-
- voluntary contributions made by Paul Eng on behalf of the
469-
Apache Software Foundation that were originally developed at iClick, Inc.,
470-
software copyright (c) 1999.
471-
472451
Apache Commons Collections
473452
Copyright 2001-2015 The Apache Software Foundation
474453

‎pom.xml

-297
Original file line numberDiff line numberDiff line change
@@ -1418,92 +1418,6 @@
14181418
<scope>test</scope>
14191419
</dependency>
14201420
<!-- End of Hadoop 3.x dependencies -->
1421-
<dependency>
1422-
<groupId>org.apache.hadoop</groupId>
1423-
<artifactId>hadoop-client</artifactId>
1424-
<version>${hadoop.version}</version>
1425-
<scope>${hadoop.deps.scope}</scope>
1426-
<exclusions>
1427-
<exclusion>
1428-
<groupId>org.fusesource.leveldbjni</groupId>
1429-
<artifactId>leveldbjni-all</artifactId>
1430-
</exclusion>
1431-
<exclusion>
1432-
<groupId>asm</groupId>
1433-
<artifactId>asm</artifactId>
1434-
</exclusion>
1435-
<exclusion>
1436-
<groupId>org.ow2.asm</groupId>
1437-
<artifactId>asm</artifactId>
1438-
</exclusion>
1439-
<exclusion>
1440-
<groupId>org.jboss.netty</groupId>
1441-
<artifactId>netty</artifactId>
1442-
</exclusion>
1443-
<exclusion>
1444-
<groupId>io.netty</groupId>
1445-
<artifactId>netty</artifactId>
1446-
</exclusion>
1447-
<exclusion>
1448-
<!-- BeanUtils >= 1.9.0 no longer splits out -core; exclude it -->
1449-
<groupId>commons-beanutils</groupId>
1450-
<artifactId>commons-beanutils-core</artifactId>
1451-
</exclusion>
1452-
<exclusion>
1453-
<groupId>commons-logging</groupId>
1454-
<artifactId>commons-logging</artifactId>
1455-
</exclusion>
1456-
<exclusion>
1457-
<groupId>org.mockito</groupId>
1458-
<artifactId>mockito-all</artifactId>
1459-
</exclusion>
1460-
<exclusion>
1461-
<groupId>org.mortbay.jetty</groupId>
1462-
<artifactId>servlet-api-2.5</artifactId>
1463-
</exclusion>
1464-
<exclusion>
1465-
<groupId>javax.servlet</groupId>
1466-
<artifactId>servlet-api</artifactId>
1467-
</exclusion>
1468-
<exclusion>
1469-
<groupId>junit</groupId>
1470-
<artifactId>junit</artifactId>
1471-
</exclusion>
1472-
<exclusion>
1473-
<groupId>com.sun.jersey</groupId>
1474-
<artifactId>*</artifactId>
1475-
</exclusion>
1476-
<exclusion>
1477-
<groupId>com.sun.jersey.jersey-test-framework</groupId>
1478-
<artifactId>*</artifactId>
1479-
</exclusion>
1480-
<exclusion>
1481-
<groupId>com.sun.jersey.contribs</groupId>
1482-
<artifactId>*</artifactId>
1483-
</exclusion>
1484-
<exclusion>
1485-
<groupId>net.java.dev.jets3t</groupId>
1486-
<artifactId>jets3t</artifactId>
1487-
</exclusion>
1488-
<!-- Hadoop-3.x -->
1489-
<exclusion>
1490-
<groupId>javax.ws.rs</groupId>
1491-
<artifactId>jsr311-api</artifactId>
1492-
</exclusion>
1493-
<exclusion>
1494-
<groupId>org.eclipse.jetty</groupId>
1495-
<artifactId>jetty-webapp</artifactId>
1496-
</exclusion>
1497-
<exclusion>
1498-
<groupId>log4j</groupId>
1499-
<artifactId>log4j</artifactId>
1500-
</exclusion>
1501-
<exclusion>
1502-
<groupId>org.slf4j</groupId>
1503-
<artifactId>slf4j-log4j12</artifactId>
1504-
</exclusion>
1505-
</exclusions>
1506-
</dependency>
15071421
<dependency>
15081422
<groupId>org.apache.hadoop</groupId>
15091423
<artifactId>hadoop-minikdc</artifactId>
@@ -1544,16 +1458,6 @@
15441458
<version>${bouncycastle.version}</version>
15451459
<scope>test</scope>
15461460
</dependency>
1547-
<!-- Managed up to match Hadoop in HADOOP-16530 -->
1548-
<!--
1549-
When upgrading `xercesImpl` version, also need to change
1550-
the version definition in `SparkBuild#DependencyOverrides`.
1551-
-->
1552-
<dependency>
1553-
<groupId>xerces</groupId>
1554-
<artifactId>xercesImpl</artifactId>
1555-
<version>2.12.2</version>
1556-
</dependency>
15571461
<dependency>
15581462
<groupId>org.apache.avro</groupId>
15591463
<artifactId>avro</artifactId>
@@ -1636,207 +1540,6 @@
16361540
<version>1.1.1</version>
16371541
<scope>${hadoop.deps.scope}</scope>
16381542
</dependency>
1639-
<dependency>
1640-
<groupId>org.apache.hadoop</groupId>
1641-
<artifactId>hadoop-yarn-api</artifactId>
1642-
<version>${yarn.version}</version>
1643-
<scope>${hadoop.deps.scope}</scope>
1644-
<exclusions>
1645-
<exclusion>
1646-
<groupId>javax.servlet</groupId>
1647-
<artifactId>servlet-api</artifactId>
1648-
</exclusion>
1649-
<exclusion>
1650-
<groupId>asm</groupId>
1651-
<artifactId>asm</artifactId>
1652-
</exclusion>
1653-
<exclusion>
1654-
<groupId>org.ow2.asm</groupId>
1655-
<artifactId>asm</artifactId>
1656-
</exclusion>
1657-
<exclusion>
1658-
<groupId>org.jboss.netty</groupId>
1659-
<artifactId>netty</artifactId>
1660-
</exclusion>
1661-
<exclusion>
1662-
<groupId>commons-logging</groupId>
1663-
<artifactId>commons-logging</artifactId>
1664-
</exclusion>
1665-
<exclusion>
1666-
<groupId>com.sun.jersey</groupId>
1667-
<artifactId>*</artifactId>
1668-
</exclusion>
1669-
<exclusion>
1670-
<groupId>com.sun.jersey.jersey-test-framework</groupId>
1671-
<artifactId>*</artifactId>
1672-
</exclusion>
1673-
<exclusion>
1674-
<groupId>com.sun.jersey.contribs</groupId>
1675-
<artifactId>*</artifactId>
1676-
</exclusion>
1677-
<exclusion>
1678-
<groupId>jdk.tools</groupId>
1679-
<artifactId>jdk.tools</artifactId>
1680-
</exclusion>
1681-
</exclusions>
1682-
</dependency>
1683-
<dependency>
1684-
<groupId>org.apache.hadoop</groupId>
1685-
<artifactId>hadoop-yarn-common</artifactId>
1686-
<version>${yarn.version}</version>
1687-
<scope>${hadoop.deps.scope}</scope>
1688-
<exclusions>
1689-
<exclusion>
1690-
<groupId>asm</groupId>
1691-
<artifactId>asm</artifactId>
1692-
</exclusion>
1693-
<exclusion>
1694-
<groupId>org.ow2.asm</groupId>
1695-
<artifactId>asm</artifactId>
1696-
</exclusion>
1697-
<exclusion>
1698-
<groupId>org.jboss.netty</groupId>
1699-
<artifactId>netty</artifactId>
1700-
</exclusion>
1701-
<exclusion>
1702-
<groupId>javax.servlet</groupId>
1703-
<artifactId>servlet-api</artifactId>
1704-
</exclusion>
1705-
<exclusion>
1706-
<groupId>commons-logging</groupId>
1707-
<artifactId>commons-logging</artifactId>
1708-
</exclusion>
1709-
<exclusion>
1710-
<groupId>com.sun.jersey</groupId>
1711-
<artifactId>*</artifactId>
1712-
</exclusion>
1713-
<exclusion>
1714-
<groupId>com.sun.jersey.jersey-test-framework</groupId>
1715-
<artifactId>*</artifactId>
1716-
</exclusion>
1717-
<exclusion>
1718-
<groupId>com.sun.jersey.contribs</groupId>
1719-
<artifactId>*</artifactId>
1720-
</exclusion>
1721-
<exclusion>
1722-
<groupId>log4j</groupId>
1723-
<artifactId>log4j</artifactId>
1724-
</exclusion>
1725-
<exclusion>
1726-
<groupId>org.slf4j</groupId>
1727-
<artifactId>slf4j-log4j12</artifactId>
1728-
</exclusion>
1729-
</exclusions>
1730-
</dependency>
1731-
<dependency>
1732-
<groupId>org.apache.hadoop</groupId>
1733-
<artifactId>hadoop-yarn-server-tests</artifactId>
1734-
<version>${yarn.version}</version>
1735-
<classifier>tests</classifier>
1736-
<scope>test</scope>
1737-
<exclusions>
1738-
<exclusion>
1739-
<groupId>org.fusesource.leveldbjni</groupId>
1740-
<artifactId>leveldbjni-all</artifactId>
1741-
</exclusion>
1742-
<exclusion>
1743-
<groupId>asm</groupId>
1744-
<artifactId>asm</artifactId>
1745-
</exclusion>
1746-
<exclusion>
1747-
<groupId>org.ow2.asm</groupId>
1748-
<artifactId>asm</artifactId>
1749-
</exclusion>
1750-
<exclusion>
1751-
<groupId>org.jboss.netty</groupId>
1752-
<artifactId>netty</artifactId>
1753-
</exclusion>
1754-
<exclusion>
1755-
<groupId>javax.servlet</groupId>
1756-
<artifactId>servlet-api</artifactId>
1757-
</exclusion>
1758-
<exclusion>
1759-
<groupId>commons-logging</groupId>
1760-
<artifactId>commons-logging</artifactId>
1761-
</exclusion>
1762-
<exclusion>
1763-
<groupId>com.sun.jersey</groupId>
1764-
<artifactId>*</artifactId>
1765-
</exclusion>
1766-
<exclusion>
1767-
<groupId>com.sun.jersey.jersey-test-framework</groupId>
1768-
<artifactId>*</artifactId>
1769-
</exclusion>
1770-
<exclusion>
1771-
<groupId>com.sun.jersey.contribs</groupId>
1772-
<artifactId>*</artifactId>
1773-
</exclusion>
1774-
<exclusion>
1775-
<groupId>org.apache.hadoop</groupId>
1776-
<artifactId>hadoop-yarn-server-resourcemanager</artifactId>
1777-
</exclusion>
1778-
</exclusions>
1779-
</dependency>
1780-
<!--
1781-
Hack to exclude org.apache.hadoop:hadoop-yarn-server-resourcemanager:jar:tests.
1782-
For some reasons, SBT starts to pull the dependencies of 'hadoop-yarn-server-tests' above
1783-
with 'tests' classifier after upgrading SBT 1.3 (SPARK-21708). Otherwise, some tests might
1784-
fail, see also SPARK-33104.
1785-
-->
1786-
<dependency>
1787-
<groupId>org.apache.hadoop</groupId>
1788-
<artifactId>hadoop-yarn-server-resourcemanager</artifactId>
1789-
<version>${yarn.version}</version>
1790-
<scope>test</scope>
1791-
</dependency>
1792-
<dependency>
1793-
<groupId>org.apache.hadoop</groupId>
1794-
<artifactId>hadoop-yarn-client</artifactId>
1795-
<version>${yarn.version}</version>
1796-
<scope>${hadoop.deps.scope}</scope>
1797-
<exclusions>
1798-
<exclusion>
1799-
<groupId>asm</groupId>
1800-
<artifactId>asm</artifactId>
1801-
</exclusion>
1802-
<exclusion>
1803-
<groupId>org.ow2.asm</groupId>
1804-
<artifactId>asm</artifactId>
1805-
</exclusion>
1806-
<exclusion>
1807-
<groupId>org.jboss.netty</groupId>
1808-
<artifactId>netty</artifactId>
1809-
</exclusion>
1810-
<exclusion>
1811-
<groupId>javax.servlet</groupId>
1812-
<artifactId>servlet-api</artifactId>
1813-
</exclusion>
1814-
<exclusion>
1815-
<groupId>commons-logging</groupId>
1816-
<artifactId>commons-logging</artifactId>
1817-
</exclusion>
1818-
<exclusion>
1819-
<groupId>com.sun.jersey</groupId>
1820-
<artifactId>*</artifactId>
1821-
</exclusion>
1822-
<exclusion>
1823-
<groupId>com.sun.jersey.jersey-test-framework</groupId>
1824-
<artifactId>*</artifactId>
1825-
</exclusion>
1826-
<exclusion>
1827-
<groupId>com.sun.jersey.contribs</groupId>
1828-
<artifactId>*</artifactId>
1829-
</exclusion>
1830-
<exclusion>
1831-
<groupId>log4j</groupId>
1832-
<artifactId>log4j</artifactId>
1833-
</exclusion>
1834-
<exclusion>
1835-
<groupId>org.slf4j</groupId>
1836-
<artifactId>slf4j-log4j12</artifactId>
1837-
</exclusion>
1838-
</exclusions>
1839-
</dependency>
18401543
<dependency>
18411544
<groupId>org.apache.zookeeper</groupId>
18421545
<artifactId>zookeeper</artifactId>

‎project/SparkBuild.scala

-1
Original file line numberDiff line numberDiff line change
@@ -1060,7 +1060,6 @@ object DependencyOverrides {
10601060
lazy val guavaVersion = sys.props.get("guava.version").getOrElse("33.1.0-jre")
10611061
lazy val settings = Seq(
10621062
dependencyOverrides += "com.google.guava" % "guava" % guavaVersion,
1063-
dependencyOverrides += "xerces" % "xercesImpl" % "2.12.2",
10641063
dependencyOverrides += "jline" % "jline" % "2.14.6",
10651064
dependencyOverrides += "org.apache.avro" % "avro" % "1.11.3")
10661065
}

‎resource-managers/kubernetes/integration-tests/pom.xml

+6-14
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,12 @@
7979
<version>${project.version}</version>
8080
<scope>test</scope>
8181
</dependency>
82+
<dependency>
83+
<groupId>software.amazon.awssdk</groupId>
84+
<artifactId>bundle</artifactId>
85+
<version>${aws.java.sdk.v2.version}</version>
86+
<scope>test</scope>
87+
</dependency>
8288
</dependencies>
8389

8490
<build>
@@ -189,20 +195,6 @@
189195
</build>
190196

191197
<profiles>
192-
<profile>
193-
<id>hadoop-3</id>
194-
<activation>
195-
<activeByDefault>true</activeByDefault>
196-
</activation>
197-
<dependencies>
198-
<dependency>
199-
<groupId>software.amazon.awssdk</groupId>
200-
<artifactId>bundle</artifactId>
201-
<version>${aws.java.sdk.v2.version}</version>
202-
<scope>test</scope>
203-
</dependency>
204-
</dependencies>
205-
</profile>
206198
<profile>
207199
<id>volcano</id>
208200
<dependencies>

0 commit comments

Comments
 (0)