Skip to content

Commit fa6bd52

Browse files
gengliangwangJackey Lee
authored and
Jackey Lee
committed
[SPARK-26076][BUILD][MINOR] Revise ambiguous error message from load-spark-env.sh
## What changes were proposed in this pull request? When I try to run scripts (e.g. `start-master.sh`/`start-history-server.sh ` in latest master, I got such error: ``` Presence of build for multiple Scala versions detected. Either clean one of them or, export SPARK_SCALA_VERSION in spark-env.sh. ``` The error message is quite confusing. Without reading `load-spark-env.sh`, I didn't know which directory to remove, or where to find and edit the `spark-evn.sh`. This PR is to make the error message more clear. Also change the script for less maintenance when we add or drop Scala versions in the future. As now with apache#22967, we can revise the error message as following(in my local setup): ``` Presence of build for multiple Scala versions detected (/Users/gengliangwang/IdeaProjects/spark/assembly/target/scala-2.12 and /Users/gengliangwang/IdeaProjects/spark/assembly/target/scala-2.11). Remove one of them or, export SPARK_SCALA_VERSION=2.12 in /Users/gengliangwang/IdeaProjects/spark/conf/spark-env.sh. Visit https://spark.apache.org/docs/latest/configuration.html#environment-variables for more details about setting environment variables in spark-env.sh. ``` ## How was this patch tested? Manual test Closes apache#23049 from gengliangwang/reviseEnvScript. Authored-by: Gengliang Wang <gengliang.wang@databricks.com> Signed-off-by: Sean Owen <sean.owen@databricks.com>
1 parent 7652db7 commit fa6bd52

File tree

1 file changed

+16
-11
lines changed

1 file changed

+16
-11
lines changed

bin/load-spark-env.sh

+16-11
Original file line numberDiff line numberDiff line change
@@ -26,35 +26,40 @@ if [ -z "${SPARK_HOME}" ]; then
2626
source "$(dirname "$0")"/find-spark-home
2727
fi
2828

29+
SPARK_ENV_SH="spark-env.sh"
2930
if [ -z "$SPARK_ENV_LOADED" ]; then
3031
export SPARK_ENV_LOADED=1
3132

3233
export SPARK_CONF_DIR="${SPARK_CONF_DIR:-"${SPARK_HOME}"/conf}"
3334

34-
if [ -f "${SPARK_CONF_DIR}/spark-env.sh" ]; then
35+
SPARK_ENV_SH="${SPARK_CONF_DIR}/${SPARK_ENV_SH}"
36+
if [[ -f "${SPARK_ENV_SH}" ]]; then
3537
# Promote all variable declarations to environment (exported) variables
3638
set -a
37-
. "${SPARK_CONF_DIR}/spark-env.sh"
39+
. ${SPARK_ENV_SH}
3840
set +a
3941
fi
4042
fi
4143

4244
# Setting SPARK_SCALA_VERSION if not already set.
4345

4446
if [ -z "$SPARK_SCALA_VERSION" ]; then
47+
SCALA_VERSION_1=2.12
48+
SCALA_VERSION_2=2.11
4549

46-
ASSEMBLY_DIR2="${SPARK_HOME}/assembly/target/scala-2.11"
47-
ASSEMBLY_DIR1="${SPARK_HOME}/assembly/target/scala-2.12"
48-
49-
if [[ -d "$ASSEMBLY_DIR2" && -d "$ASSEMBLY_DIR1" ]]; then
50-
echo -e "Presence of build for multiple Scala versions detected." 1>&2
51-
echo -e 'Either clean one of them or, export SPARK_SCALA_VERSION in spark-env.sh.' 1>&2
50+
ASSEMBLY_DIR_1="${SPARK_HOME}/assembly/target/scala-${SCALA_VERSION_1}"
51+
ASSEMBLY_DIR_2="${SPARK_HOME}/assembly/target/scala-${SCALA_VERSION_2}"
52+
ENV_VARIABLE_DOC="https://spark.apache.org/docs/latest/configuration.html#environment-variables"
53+
if [[ -d "$ASSEMBLY_DIR_1" && -d "$ASSEMBLY_DIR_2" ]]; then
54+
echo "Presence of build for multiple Scala versions detected ($ASSEMBLY_DIR_1 and $ASSEMBLY_DIR_2)." 1>&2
55+
echo "Remove one of them or, export SPARK_SCALA_VERSION=$SCALA_VERSION_1 in ${SPARK_ENV_SH}." 1>&2
56+
echo "Visit ${ENV_VARIABLE_DOC} for more details about setting environment variables in spark-env.sh." 1>&2
5257
exit 1
5358
fi
5459

55-
if [ -d "$ASSEMBLY_DIR2" ]; then
56-
export SPARK_SCALA_VERSION="2.11"
60+
if [[ -d "$ASSEMBLY_DIR_1" ]]; then
61+
export SPARK_SCALA_VERSION=${SCALA_VERSION_1}
5762
else
58-
export SPARK_SCALA_VERSION="2.12"
63+
export SPARK_SCALA_VERSION=${SCALA_VERSION_2}
5964
fi
6065
fi

0 commit comments

Comments
 (0)