Skip to content

Commit aee5d70

Browse files
sarutakdbtsai
authored andcommitted
[SPARK-34762][BUILD] Fix the build failure with Scala 2.13 which is related to commons-cli
### What changes were proposed in this pull request? This PR fixes the build failure with Scala 2.13 which is related to `commons-cli`. The last few days, build with Scala 2.13 on GA continues to fail and the error message says like as follows. ``` [error] /home/runner/work/spark/spark/sql/hive-thriftserver/src/main/java/org/apache/hive/service/server/HiveServer2.java:26:1: error: package org.apache.commons.cli does not exist 1278[error] import org.apache.commons.cli.GnuParser; ``` The reason is that `mvn help` in `change-scala-version.sh` downloads the POM file of `commons-cli` but doesn't download the JAR file, leading the build failure. This PR also adds `commons-cli` to the dependencies explicitly because HiveThriftServer depends on it. ### Why are the changes needed? Expect to fix the build failure with Scala 2.13. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? I confirmed that build successfully finishes with Scala 2.13 on my laptop. ``` find ~/.m2 -name commons-cli -exec rm -rf {} \; find ~/.ivy2 -name commons-cli -exec rm -rf {} \; find ~/.cache/ -name commons-cli -exec rm -rf {} \; // For Linux find ~/Library/Caches -name commons-cli -exec rm -rf {} \; // For macOS dev/change-scala-version 2.13 ./build/sbt -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-cloud -Pkinesis-asl -Pdocker-integration-tests -Pkubernetes-integration-tests -Pspark-ganglia-lgpl -Pscala-2.13 clean compile test:compile ``` Closes apache#31862 from sarutak/commons-cli. Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> (cherry picked from commit c5cadfe) Signed-off-by: HyukjinKwon <gurwls223@apache.org>
1 parent 43390d9 commit aee5d70

File tree

3 files changed

+15
-0
lines changed

3 files changed

+15
-0
lines changed

dev/change-scala-version.sh

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,11 @@ BASEDIR=$(dirname $0)/..
6060
find "$BASEDIR" -name 'pom.xml' -not -path '*target*' -print \
6161
-exec bash -c "sed_i 's/\(artifactId.*\)_'$FROM_VERSION'/\1_'$TO_VERSION'/g' {}" \;
6262

63+
# dependency:get is workaround for SPARK-34762 to download the JAR file of commons-cli.
64+
# Without this, build with Scala 2.13 using SBT will fail because the help plugin used below downloads only the POM file.
65+
COMMONS_CLI_VERSION=`build/mvn help:evaluate -Dexpression=commons-cli.version -q -DforceStdout`
66+
build/mvn dependency:get -Dartifact=commons-cli:commons-cli:${COMMONS_CLI_VERSION} -q
67+
6368
# Update <scala.version> in parent POM
6469
# First find the right full version from the profile's build
6570
SCALA_VERSION=`build/mvn help:evaluate -Pscala-${TO_VERSION} -Dexpression=scala.version -q -DforceStdout`

pom.xml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -200,6 +200,7 @@
200200
<paranamer.version>2.8</paranamer.version>
201201
<maven-antrun.version>1.8</maven-antrun.version>
202202
<commons-crypto.version>1.1.0</commons-crypto.version>
203+
<commons-cli.version>1.2</commons-cli.version>
203204
<!--
204205
If you are changing Arrow version specification, please check
205206
./python/pyspark/sql/pandas/utils.py, and ./python/setup.py too.
@@ -2400,6 +2401,11 @@
24002401
</exclusion>
24012402
</exclusions>
24022403
</dependency>
2404+
<dependency>
2405+
<groupId>commons-cli</groupId>
2406+
<artifactId>commons-cli</artifactId>
2407+
<version>${commons-cli.version}</version>
2408+
</dependency>
24032409
</dependencies>
24042410
</dependencyManagement>
24052411

sql/hive-thriftserver/pom.xml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,10 @@
133133
<groupId>net.sf.jpam</groupId>
134134
<artifactId>jpam</artifactId>
135135
</dependency>
136+
<dependency>
137+
<groupId>commons-cli</groupId>
138+
<artifactId>commons-cli</artifactId>
139+
</dependency>
136140
</dependencies>
137141
<build>
138142
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>

0 commit comments

Comments
 (0)