Building Apache Zeppelin

Building Apache Zeppelin

The instructions provided below specify the steps to build Apache Zeppelin version 0.8.1 on Linux on IBM Z for the following distributions:

• RHEL (7.5, 7.6)
• Ubuntu (16.04, 18.04)

General Notes:

• When following the steps below please use a standard permission user unless otherwise specified.
• A directory /<source_root>/ will be referred to in these instructions, this is a temporary writeable directory anywhere you'd like to place it.

Step 1 : Install Dependencies

export SOURCE_ROOT=/<source_root>/

1.1) Install packages

• RHEL (7.5, 7.6)

  sudo yum install git tar wget java-1.8.0-openjdk-headless java-1.8.0-openjdk-devel
  

• Ubuntu (16.04, 18.04)

  sudo apt-get update
  sudo apt-get install build-essential git tar wget maven openjdk-8-jdk-headless openjdk-8-jdk python
  

1.2 ) Download Maven binary executable (Only for RHEL)

cd $SOURCE_ROOT
wget http://www.eu.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
sudo tar -zxf apache-maven-3.3.9-bin.tar.gz -C /usr/local/
sudo ln -s /usr/local/apache-maven-3.3.9/bin/mvn /usr/local/bin/mvn

1.3) Set environment variable

export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=1024m"
export JAVA_HOME=/usr/lib/jvm/java                   # RHEL (7.5, 7.6)
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-s390x   # Ubuntu (16.04, 18.04)
export PATH=$JAVA_HOME/bin:$PATH

1.4) Install PhantomJS 2.1.1

• Follow the PhantomJS recipe. For Ubuntu, install using sudo apt-get install phantomjs. Make sure PhantomJS is available in PATH environment variable.

1.5) Install Protobuf 3.3.0

• Follow the ProtoBuf build instructions.

1.6) Install gRPC-Java 1.4.0

cd $SOURCE_ROOT
git clone https://github.com/grpc/grpc-java.git
cd grpc-java/
git checkout v1.4.0

• Edit $SOURCE_ROOT/grpc-java/compiler/build.gradle

  @@ -49,6 +49,9 @@
       gcc(Gcc) {
         target("ppcle_64")
       }
  +    gcc(Gcc) {
  +      target("s390_64")
  +    }
       clang(Clang) {
       }
     }
  @@ -63,11 +66,14 @@
       ppcle_64 {
         architecture "ppcle_64"
       }
  +    s390_64 {
  +      architecture "s390_64"
  +    }
     }
   
     components {
       java_plugin(NativeExecutableSpec) {
  -      if (arch in ['x86_32', 'x86_64', 'ppcle_64']) {
  +      if (arch in ['x86_32', 'x86_64', 'ppcle_64', 's390_64']) {
           // If arch is not within the defined platforms, we do not specify the
           // targetPlatform so that Gradle will choose what is appropriate.
           targetPlatform arch  

• Build gRPC-java

cd $SOURCE_ROOT/grpc-java
./gradlew install -Pprotoc=/usr/local/bin/protoc  

Note: While building gRPC-Java, if the build fails with an error grpc-compiler:linkJava_pluginExecutable/bin/ld: cannot find -lstdc++, install libstdc++-static

Step 2 : Build and Install Apache Zeppelin

2.1) Download source

git clone https://github.com/apache/zeppelin.git
cd zeppelin/
git checkout v0.8.1

2.2) Create a patch file $SOURCE_ROOT/zeppelin/zepp_patch.diff with the following contents:

diff --git a/pom.xml b/pom.xml
index 61646d6..df60339 100644
--- a/pom.xml
+++ b/pom.xml
@@ -95,7 +95,7 @@
     <!-- frontend maven plugin related versions-->
     <node.version>v8.9.3</node.version>
     <npm.version>5.5.1</npm.version>
-    <plugin.frontend.version>1.4</plugin.frontend.version>
+    <plugin.frontend.version>1.5</plugin.frontend.version>
 
     <!-- common library versions -->
     <slf4j.version>1.7.10</slf4j.version>

Apply patch using command mentioned below:

    patch $SOURCE_ROOT/zeppelin/pom.xml < $SOURCE_ROOT/zeppelin/zepp_patch.diff

2.3) Build Apache Zeppelin

cd $SOURCE_ROOT/zeppelin/
mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.3.0 -Dclassifier=linux-s390_64 -Dpackaging=exe -Dfile=$SOURCE_ROOT/protobuf/src/.libs/protoc
mvn install:install-file -DgroupId=io.grpc -DartifactId=protoc-gen-grpc-java -Dversion=1.4.0 -Dclassifier=linux-s390_64 -Dpackaging=exe -Dfile=$SOURCE_ROOT/grpc-java/compiler/build/exe/java_plugin/protoc-gen-grpc-java
mvn clean package -DskipTests

Note: If an error of QXchConnection: Could not connect to display occurs, run export QT_QPA_PLATFORM=offscreen

Step 3 : Testing (Optional)

3.1) For resolving Cassandra test failure, follow below mentioned steps:

• Create a patch file $SOURCE_ROOT/zeppelin/cassandra/cass_patch.diff with the following contents:

diff --git a/cassandra/pom.xml b/cassandra/pom.xml
index a8f198b..3b2e27c 100644
--- a/cassandra/pom.xml
+++ b/cassandra/pom.xml
@@ -43,7 +43,7 @@
 
         <!-- test library versions -->
         <achilles.version>3.2.4-Zeppelin</achilles.version>
-        <jna.version>4.2.0</jna.version>
+        <jna.version>4.5.0</jna.version>
 
         <!-- plugin versions -->
         <plugin.scala.version>2.15.2</plugin.scala.version>

Apply patch using command mentioned below:

	patch $SOURCE_ROOT/zeppelin/cassandra/pom.xml < $SOURCE_ROOT/zeppelin/cassandra/cass_patch.diff

• Build Achilles from source to increase server wait time and replace achillles-embedded jar:

cd $SOURCE_ROOT/zeppelin/
git clone https://github.com/doanduyhai/Achilles.git
cd Achilles/
git checkout 3.2-Zeppelin
sed -i -e '141s/30/600/' $SOURCE_ROOT/zeppelin/Achilles/achilles-embedded/src/main/java/info/archinnov/achilles/embedded/ServerStarter.java
mvn clean package -DskipTests 
cp $SOURCE_ROOT/zeppelin/Achilles/achilles-embedded/target/Achilles.jar $SOURCE_ROOT/.m2/repository/info/archinnov/achilles-embedded/3.2.4-Zeppelin/achilles-embedded-3.2.4-Zeppelin.jar

3.2) Run the test cases :

cd $SOURCE_ROOT/zeppelin/
mvn package -fn

Note:

• Individual test case can be executed by running command mvn clean test in the respective modules

• Test case failures in modules Zengine, Python interpeter , Spark Interpreter and Server can be ignored as they are seen on x86 system as well

Step 4 : Start Apache Zeppelin

cd $SOURCE_ROOT/zeppelin/
bin/zeppelin-daemon.sh start 

Open http://<ip_address>:8080/ in your browser to access Web UI.

References

• https://zeppelin.apache.org/