Skip to content

[ZEPPELIN-1256][BUILD] Build distribution package with Spark 2.0 and Scala 2.11 #1251

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
minahlee wants to merge 3 commits into from
Closed

[ZEPPELIN-1256][BUILD] Build distribution package with Spark 2.0 and Scala 2.11 #1251

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4ca3e7f
Build distribution package with Spark 2.0 and Scala 2.11
minahlee Aug 2, 2016
99d91bf
Update interpreter installation for the user who use Spark version <=…
minahlee Aug 2, 2016
ff034c6
change scala-2.10 and scala-2.11 profile activation rule
minahlee Aug 5, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .travis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,9 @@ addons:

matrix:
include:
# Test all modules with spark-2.0.0-preview and scala 2.11
# Test all modules with spark 2.0.0 and scala 2.11
- jdk: "oraclejdk7"
env: SCALA_VER="2.11" SPARK_VER="2.0.0" HADOOP_VER="2.3" PROFILE="-Pspark-2.0 -Dspark.version=2.0.0-preview -Phadoop-2.3 -Ppyspark -Psparkr -Pscalding -Pexamples -Pscala-2.11" BUILD_FLAG="package -Pbuild-distr" TEST_FLAG="verify -Pusing-packaged-distr" TEST_PROJECTS=""
env: SCALA_VER="2.11" SPARK_VER="2.0.0" HADOOP_VER="2.3" PROFILE="-Pspark-2.0 -Phadoop-2.3 -Ppyspark -Psparkr -Pscalding -Pexamples -Pscala-2.11" BUILD_FLAG="package -Pbuild-distr" TEST_FLAG="verify -Pusing-packaged-distr" TEST_PROJECTS=""

# Test all modules with scala 2.10
- jdk: "oraclejdk7"
Expand Down
2 changes: 2 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,7 @@ Here're some examples:

```sh
# build with spark-2.0, scala-2.11
./dev/change_scala_version.sh 2.11
mvn clean package -Pspark-2.0 -Phadoop-2.4 -Pyarn -Ppyspark -Psparkr -Pscala-2.11

# build with spark-1.6, scala-2.10
Expand Down Expand Up @@ -306,6 +307,7 @@ For configuration details check __`./conf`__ subdirectory.
To produce a Zeppelin package compiled with Scala 2.11, use the -Pscala-2.11 profile:

```
./dev/change_scala_version.sh 2.11
mvn clean package -Pspark-1.6 -Phadoop-2.4 -Pyarn -Ppyspark -Pscala-2.11 -DskipTests clean install
```

Expand Down
5 changes: 3 additions & 2 deletions dev/create_release.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ function make_binary_release() {

cp -r "${WORKING_DIR}/zeppelin" "${WORKING_DIR}/zeppelin-${RELEASE_VERSION}-bin-${BIN_RELEASE_NAME}"
cd "${WORKING_DIR}/zeppelin-${RELEASE_VERSION}-bin-${BIN_RELEASE_NAME}"
./dev/change_scala_version.sh 2.11
echo "mvn clean package -Pbuild-distr -DskipTests ${BUILD_FLAGS}"
mvn clean package -Pbuild-distr -DskipTests ${BUILD_FLAGS}
if [[ $? -ne 0 ]]; then
Expand Down Expand Up @@ -102,8 +103,8 @@ function make_binary_release() {

git_clone
make_source_package
make_binary_release all "-Pspark-1.6 -Phadoop-2.4 -Pyarn -Ppyspark -Psparkr -Pr"
make_binary_release netinst "-Pspark-1.6 -Phadoop-2.4 -Pyarn -Ppyspark -Psparkr -Pr -pl !alluxio,!angular,!cassandra,!elasticsearch,!file,!flink,!hbase,!ignite,!jdbc,!kylin,!lens,!livy,!markdown,!postgresql,!python,!shell"
make_binary_release all "-Pspark-2.0 -Phadoop-2.4 -Pyarn -Ppyspark -Psparkr -Pr -Pscala-2.11"
make_binary_release netinst "-Pspark-2.0 -Phadoop-2.4 -Pyarn -Ppyspark -Psparkr -Pr -Pscala-2.11 -pl !alluxio,!angular,!cassandra,!elasticsearch,!file,!flink,!hbase,!ignite,!jdbc,!kylin,!lens,!livy,!markdown,!postgresql,!python,!shell"

# remove non release files and dirs
rm -rf "${WORKING_DIR}/zeppelin"
Expand Down
3 changes: 2 additions & 1 deletion docs/install/install.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@ Here are some examples with several options

```
# build with spark-2.0, scala-2.11
./dev/change_scala_version.sh 2.11
mvn clean package -Pspark-2.0 -Phadoop-2.4 -Pyarn -Ppyspark -Psparkr -Pscala-2.11

# build with spark-1.6, scala-2.10
Expand Down Expand Up @@ -394,4 +395,4 @@ You can configure Apache Zeppelin with both **environment variables** in `conf/z
<td>1024000</td>
<td>Size in characters of the maximum text message to be received by websocket.</td>
</tr>
</table>
</table>
9 changes: 9 additions & 0 deletions docs/manual/interpreterinstallation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,15 @@ If you install one of these interpreters only with `--name` option, installer wi
./bin/install-interpreter.sh --name flink --artifact org.apache.zeppelin:zeppelin-flink_2.10:0.6.1
```

#### Install Spark interpreter built with Scala 2.10
Spark distribution package has been built with Scala 2.10 until 1.6.2. If you have `SPARK_HOME` set pointing to Spark version ealier than 2.0.0, you need to download Spark interpreter packaged with Scala 2.10. To do so, use follow command:

```
rm -rf ./interpreter/spark
./bin/install-interpreter.sh --name spark --artifact org.apache.zeppelin:zeppelin-spark_2.10:0.6.1
```

<br />
Once you have installed interpreters, you need to restart Zeppelin. And then [create interpreter setting](../manual/interpreters.html#what-is-zeppelin-interpreter) and [bind it with your notebook](../manual/interpreters.html#what-is-zeppelin-interpreter-setting).


Expand Down
5 changes: 1 addition & 4 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -652,7 +652,7 @@
<profile>
<id>scala-2.10</id>
<activation>
<property><name>!scala-2.11</name></property>
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<scala.version>2.10.5</scala.version>
Expand All @@ -662,9 +662,6 @@

<profile>
<id>scala-2.11</id>
<activation>
<property><name>scala-2.11</name></property>
</activation>
<properties>
<scala.version>2.11.7</scala.version>
<scala.binary.version>2.11</scala.binary.version>
Expand Down
5 changes: 1 addition & 4 deletions r/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -380,7 +380,7 @@
<profile>
<id>scala-2.10</id>
<activation>
<property><name>!scala-2.11</name></property>
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<extra.source.dir>src/main/scala-2.10</extra.source.dir>
Expand All @@ -390,9 +390,6 @@

<profile>
<id>scala-2.11</id>
<activation>
<property><name>scala-2.11</name></property>
</activation>
<properties>
<extra.source.dir>src/main/scala-2.11</extra.source.dir>
<extra.testsource.dir>src/test/scala/scala-2.11</extra.testsource.dir>
Expand Down
4 changes: 0 additions & 4 deletions zeppelin-display/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,10 +94,6 @@
<profiles>
<profile>
<id>scala-2.11</id>
<activation>
<property><name>scala-2.11</name></property>
</activation>

<dependencies>
<dependency>
<groupId>org.scala-lang.modules</groupId>
Expand Down
4 changes: 0 additions & 4 deletions zeppelin-distribution/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,10 +114,6 @@
<profiles>
<profile>
<id>scala-2.11</id>
<activation>
<property><name>scala-2.11</name></property>
</activation>

<dependencyManagement>
<dependencies>
<dependency>
Expand Down
4 changes: 0 additions & 4 deletions zeppelin-server/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -413,10 +413,6 @@
<profiles>
<profile>
<id>scala-2.11</id>
<activation>
<property><name>scala-2.11</name></property>
</activation>

<dependencyManagement>
<dependencies>
<dependency>
Expand Down