#1612 Separating Menas UI and API (#1620)

* 1422 and 1423 Remove HDFS and Oozie from Menas

* #1422 Fix HDFS location validation

* #1424 Add Menas Dockerfile

* #1416 hadoop-aws 2.8.5 + s3 aws sdk 2.13.65 compiles.

* #1416 - enceladus on S3:

 - all directly-hdfs touching stuff disabled (atum, performance measurements, info files, output path checking)

# Add menasfargate into hosts
sudo nano /etc/hosts
# paste
20.0.63.69 menasfargate
# save & exit (ctrl+O, ctrl+X)

# Running standardization works via:
spark-submit --class za.co.absa.enceladus.standardization.StandardizationJob --conf "spark.driver.extraJavaOptions=-Dmenas.rest.uri=http://menasfargate:8080 -Dstandardized.hdfs.path=s3://euw1-ctodatadev-dev-bigdatarnd-s3-poc/enceladusPoc/ao-hdfs-data/stdOutput/standardized-{0}-{1}-{2}-{3}" ~/enceladusPoc/spark-jobs-2.11.0-SNAPSHOT.jar --menas-credentials-file ~/enceladusPoc/menas-credentials.properties --dataset-name dk_test1_emr285 --raw-format json --dataset-version 1 --report-date 2019-11-27 --report-version 1 2> ~/enceladusPoc/stderr.txt

* #1416 - enceladus on S3 - (crude) conformance works on s3 (s3 std input, s3 conf output)

 0- all directly-hdfs touching stuff disabled (atum, performance measurements, info files, output path checking)

# Add menasfargate into hosts
sudo nano /etc/hosts
# paste
20.0.63.69 menasfargate
# save & exit (ctrl+O, ctrl+X)

# Running conformance works via:
spark-submit --class za.co.absa.enceladus.conformance.DynamicConformanceJob --conf "spark.driver.extraJavaOptions=-Dmenas.rest.uri=http://menasfargate:8080 -Dstandardized.hdfs.path=s3://euw1-ctodatadev-dev-bigdatarnd-s3-poc/enceladusPoc/ao-hdfs-data/stdOutput/standardized-{0}-{1}-{2}-{3}" ~/enceladusPoc/spark-jobs-2.11.0-SNAPSHOT.jar --menas-credentials-file ~/enceladusPoc/menas-credentials.properties --dataset-name dk_test1_emr285 --dataset-version 1 --report-date 2019-11-27 --report-version 1 2> ~/enceladusPoc/conf-log.txt

* ref issue = 1416

* related test cases ignored (issue reference added)

* PR updates

* Merge spline 0.5.3 into aws-poc

* Update spline to 0.5.4 for AWS PoC

* #1503 Remove HDFS url Validation

This is a temporary solution. We currently experiment with
many forms of URLs, and having a regex there now slows us down.

* New dockerfile - smaller image

* s3 persistence (atum, sdk fs usage, ...) (#1526)

#1526 
* FsUtils divided into LocalFsUtils & HdfsUtils
* PathConfigSuite update
* S3FsUtils with tail-recursive pagination accumulation - now generic with optional short-circuit breakOut
TestRunnerJob updated to manually cover the cases - should serve as a basis for tests
* HdfsUtils replace by trait DistributedFsUtils (except for MenasCredentials loading & nonSplittable splitting)
* using final version of s3-powered Atum (3.0.0)
* mockito-update version update, scalatest version update
* S3FsUtilsSuite: exists, read, sizeDir(hidden, non-hidden, reucursive), non-splittable (simple, recursive with breakOut), delete (recursive), version find (simple - empty, recursive)
* explicit stubbing fix for hyperdrive

* Feature/1556 file access PoC using Hadoop FS API (#1586)

* s3 using hadoop fs api
* s3 sdk usage removed (pom, classes, tests)
* atum final version 3.1.0 used
* readStandardizationInputData(... path: String)(implicit ... fs: FileSystem) -> readStandardizationInputData(input: PathWithFs)

* 1554 Tomcat with TLS in Docker container (#1585)

* #1554 Tomcat with TLS container

* #1554 Added envoy config + enabling running unencrypted container

* #1499 Add authentication to /lineage + update spline to 0.5.5

* #1618 - fixes failing spline 0.5.5 integration by providing compatible commons library version. Test-ran on EMR. (#1619)

* #1612 Separation start

* #1612 Updated DAO for spark-jobs

* #1612 Fixed spline integration and schema, removed redundant code

* #1612 Fixed tests, removed unused dependency

* #1612 Added back dependency

* WIP fixing merge issues

* * Merge compiles
* Tests pass
* Depends on ATUM 3.1.1-SNAPSHOT (the bugfix for AbsaOSS/atum#48)

* #1612 Removed Spring from menas-web, enabled building war and static resources. Removed version subpath in menas-web + added theme dependencies in repo

* #1612 Cookies + updated lineage

* * put back HDFS browser
* put back Oozie
* downgraded Spline

* * AWS SDK Exclusion

* #1612 Included HDFSFolder + missing Oozie parts

* * New ATUM version

* * Adding missing files

* #1612 menas-web on nginx container and passing API_URL

* #1612 Working TLS on nginx, resources not included in code

* 1622: Merge of aws-poc to develop brach
* Addressed issues identified by reviewers

* * comments improvement

* 1434 Add new way of serving properties to Docker

* #1612 Building using ui5 + reused /api route

* #1612 Project version

* #713 Add favicon

* #1612 Merges

* #1612 pom parent version

* #1648 Fix war deployment + adding back spline to menas

* #1612 other fixes

* #1612 added pom package.json version sync

* #1612 newline

* #1612 fix version sync + cleaning dist

* 1648 merge to develop

* 1648 merge fix

* 1648 Fixes schema upload

* 1648 Fixes schema registry request

* 1648 pom version

* 1612 add docker build

* #601 Swagger 2 PoC

* #601 Swagger 2 PoC

* #601 Swagger 2 PoC

* #1648 Updating menas-web to 3.0

* #1612 Updated npm project versions + mvn plugin

* #1612 license_check.yml

* #1612 licence check fix

Co-authored-by: Saša Zejnilović <zejnils@gmail.com>
Co-authored-by: Daniel Kavan <dk1844@gmail.com>
Co-authored-by: Jan Scherbaum <kmoj02@gmail.com>
Co-authored-by: David Benedeki <benedeki@volny.cz>

.github/workflows/license_check.yml

@@ -0,0 +1,31 @@
+#
+# Copyright 2018 ABSA Group Limited
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+name: License Check
+
+on:
+  pull_request:
+    types: [opened, synchronize]
+
+jobs:
+  license-test:
+    name: License Check  
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-java@v1
+        with:
+          java-version: 1.8
+      - run: mvn -Plicense-check apache-rat:check

dao/src/main/scala/za/co/absa/enceladus/dao/rest/AuthClient.scala

@@ -71,14 +71,14 @@ sealed abstract class AuthClient(username: String, restTemplate: RestTemplate, a
 
   private def getAuthHeaders(response: ResponseEntity[String]): HttpHeaders = {
     val headers = response.getHeaders
-    val sessionCookie = headers.get("set-cookie").asScala.head
+    val jwt = headers.get("JWT").asScala.head
     val csrfToken = headers.get("X-CSRF-TOKEN").asScala.head
 
-    log.info(s"Session Cookie: $sessionCookie")
+    log.info(s"JWT: $jwt")
     log.info(s"CSRF Token: $csrfToken")
 
     val resultHeaders = new HttpHeaders()
-    resultHeaders.add("cookie", sessionCookie)
+    resultHeaders.add("JWT", jwt)
     resultHeaders.add("X-CSRF-TOKEN", csrfToken)
     resultHeaders
   }

dao/src/test/scala/za/co/absa/enceladus/dao/rest/auth/AuthClientSuite.scala

@@ -44,17 +44,17 @@ abstract class AuthClientSuite() extends AnyWordSpec
 
   s"Calling authenticate()" should {
     "return authentication headers on 200 OK" in {
-      val sessionCookie = "session-cookie"
+      val jwt = "jwt"
       val csrfToken = "csrf-token"
 
       val responseHeaders = new LinkedMultiValueMap[String, String]
-      responseHeaders.add("set-cookie", sessionCookie)
+      responseHeaders.add("jwt", jwt)
       responseHeaders.add("x-csrf-token", csrfToken)
 
       setUpSuccessfulAuthRequest(responseHeaders)
 
       val expected = new HttpHeaders()
-      expected.add("cookie", sessionCookie)
+      expected.add("jwt", jwt)
       expected.add("x-csrf-token", csrfToken)
 
       val response = authClient.authenticate()

menas-web/Dockerfile

@@ -0,0 +1,26 @@
+# Copyright 2018 ABSA Group Limited
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM nginx:1-alpine
+
+LABEL \
+    vendor="AbsaOSS" \
+    copyright="2020 ABSA Group Limited" \
+    license="Apache License, version 2.0" \
+    name="Menas Web"
+
+ADD start_menas_web.sh /docker-entrypoint.d/start_menas_web.sh
+COPY nginx.conf nginx.conf
+RUN chmod +x /docker-entrypoint.d/start_menas_web.sh && rm -rf /usr/share/nginx/html/*
+
+COPY ./ui/dist /usr/share/nginx/html/

menas-web/nginx.conf

@@ -0,0 +1,27 @@
+# Copyright 2018 ABSA Group Limited
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+events {}
+
+http {
+    include mime.types;
+    ssl_session_cache   shared:SSL:10m;
+    ssl_session_timeout 10m;
+    server {
+        listen              443 ssl http2;
+        root                /usr/share/nginx/html;
+        server_name         ${DNS_NAME};
+        ssl_certificate     /etc/ssl/certificate.pem;
+        ssl_certificate_key /etc/ssl/private.pem;
+    }
+}

menas-web/pom.xml

@@ -0,0 +1,154 @@
+<!--
+  ~ Copyright 2018 ABSA Group Limited
+  ~
+  ~ Licensed under the Apache License, Version 2.0 (the "License");
+  ~ you may not use this file except in compliance with the License.
+  ~ You may obtain a copy of the License at
+  ~     http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  -->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+    <artifactId>menas-web</artifactId>
+    <packaging>war</packaging>
+
+    <parent>
+        <groupId>za.co.absa.enceladus</groupId>
+        <artifactId>parent</artifactId>
+        <version>3.0.0-SNAPSHOT</version>
+    </parent>
+
+    <build>
+        <plugins>
+            <plugin>
+                <groupId>com.spotify</groupId>
+                <artifactId>dockerfile-maven-plugin</artifactId>
+                <version>${dockerfile.maven.version}</version>
+                <configuration>
+                    <repository>${dockerfile.menas-ui.repository}</repository>
+                    <tag>${dockerfile.tag}</tag>
+                </configuration>
+            </plugin>
+                <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-clean-plugin</artifactId>
+                <version>3.1.0</version>
+                <configuration>
+                    <filesets>
+                        <fileset>
+                            <directory>${basedir}/ui/dist</directory>
+                            <directory>${basedir}/ui/node_modules</directory>
+                        </fileset>
+                    </filesets>
+                </configuration>
+            </plugin>
+            <plugin>
+                <groupId>com.github.eirslett</groupId>
+                <artifactId>frontend-maven-plugin</artifactId>
+                <version>1.7.6</version>
+                <configuration>
+                    <installDirectory>target</installDirectory>
+                    <workingDirectory>ui</workingDirectory>
+                    <nodeVersion>v10.16.0</nodeVersion>
+                </configuration>
+                <executions>
+                    <execution>
+                        <id>install node and npm</id>
+                        <goals>
+                            <goal>install-node-and-npm</goal>
+                        </goals>
+                    </execution>
+                    <execution>
+                        <id>install</id>
+                        <goals>
+                            <goal>npm</goal>
+                        </goals>
+                        <configuration>
+                            <arguments>install</arguments>
+                        </configuration>
+                    </execution>
+                    <execution>
+                        <id>sync version to package.json</id>
+                        <goals>
+                            <goal>npm</goal>
+                        </goals>
+                        <configuration>
+                            <arguments>version ${project.parent.version} --allow-same-version</arguments>
+                        </configuration>
+                    </execution>
+                    <execution>
+                        <id>ui5 build</id>
+                        <goals>
+                            <goal>npm</goal>
+                        </goals>
+                        <configuration>
+                            <arguments>run ui5-build</arguments>
+                        </configuration>
+                    </execution>
+                    <execution>
+                        <id>clean node_modules</id>
+                        <goals>
+                            <goal>npm</goal>
+                        </goals>
+                        <configuration>
+                            <arguments>prune --production</arguments>
+                        </configuration>
+                    </execution>
+                </executions>
+            </plugin>
+
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-antrun-plugin</artifactId>
+                <version>${maven.ant.plugin.version}</version>
+                <executions>
+                    <execution>
+                        <id>Clean up after generating component preload</id>
+                        <phase>generate-resources</phase>
+                        <configuration>
+                            <target>
+                                <delete dir="${project.basedir}/ui/dist/dist"/>
+                                <delete dir="${project.basedir}/ui/node"/>
+                                <move toDir="${project.basedir}/ui/dist/node_modules">
+                                    <fileset dir="${project.basedir}/ui/node_modules/" includes="**/*" />
+                                </move>
+                            </target>
+                        </configuration>
+                        <goals>
+                            <goal>run</goal>
+                        </goals>
+                    </execution>
+                </executions>
+            </plugin>
+
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-war-plugin</artifactId>
+                <version>${maven.war.version}</version>
+                <configuration>
+                    <failOnMissingWebXml>false</failOnMissingWebXml>
+                    <webResources>
+                        <resource>
+                            <directory>ui/dist</directory>
+                            <targetPath>/</targetPath>
+                            <filtering>false</filtering>
+                        </resource>
+                    </webResources>
+                    <packagingExcludes>META-INF/*.SF, META-INF/*.DSA, META-INF/*.RSA</packagingExcludes>
+                    <archive>
+                        <manifest>
+                            <addDefaultImplementationEntries>true</addDefaultImplementationEntries>
+                        </manifest>
+                    </archive>
+                </configuration>
+            </plugin>
+        </plugins>
+    </build>
+
+</project>

menas-web/start_menas_web.sh

@@ -0,0 +1,25 @@
+#!/bin/sh
+
+# Copyright 2018 ABSA Group Limited
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+if [ -n "${PRIVATE_KEY}" ] && [ -n "${CERTIFICATE}" ]; then
+  envsubst < ./nginx.conf > /etc/nginx/nginx.conf
+  echo "${PRIVATE_KEY}" >> /etc/ssl/private.pem
+  echo "${CERTIFICATE}" >> /etc/ssl/certificate.pem
+  echo "${CA_CHAIN}" >> /etc/ssl/cachain.pem
+fi
+
+envsubst < /usr/share/nginx/html/package.json > /usr/share/nginx/html/package-new.json
+rm /usr/share/nginx/html/package.json
+mv /usr/share/nginx/html/package-new.json /usr/share/nginx/html/package.json

menas/ui/components/login/loginDetail.controller.js → menas-web/ui/components/login/loginDetail.controller.js

@@ -22,6 +22,12 @@ sap.ui.define([
   const usernameField = "username";
   const passwordField = "password";
 
+  function setCookie(name, value, days) {
+    let d = new Date;
+    d.setTime(d.getTime() + 24*60*60*1000*days);
+    document.cookie = name + "=" + value + ";path=/;expires=" + d.toGMTString();
+  }
+
   return Controller.extend("components.login.loginDetail", {
     loginForm: {},
 
@@ -124,8 +130,11 @@ sap.ui.define([
       let fnSuccess = (result, status, xhr) => {
         this.byId("password").setValue("");
         let csrfToken = xhr.getResponseHeader("X-CSRF-TOKEN");
+        let jwt = xhr.getResponseHeader("JWT");
         localStorage.setItem("csrfToken", csrfToken);
-        Functions.ajax("api/user/info", "GET", {}, (oInfo) => {
+        localStorage.setItem("jwtToken", jwt);
+        setCookie("JWT", jwt, 1);
+        Functions.ajax("/user/info", "GET", {}, (oInfo) => {
           model.setProperty("/userInfo", oInfo);
           model.setProperty("/menasVersion", oInfo.menasVersion);
           sap.ui.getCore().byId(this._appId).backToTopMaster();
@@ -140,7 +149,7 @@ sap.ui.define([
         this.byId(usernameField).setValueState(sap.ui.core.ValueState.Error);
         this.byId(passwordField).setValueState(sap.ui.core.ValueState.Error);
       };
-      $.ajax("api/login", {
+      $.ajax(window.apiUrl + "/login", {
         complete: function () {
           if (oControl) oControl.setBusy(false)
         },

menas/ui/components/schema/schemaDetail.controller.js → menas-web/ui/components/schema/schemaDetail.controller.js

@@ -47,6 +47,8 @@ sap.ui.define([
 
       this._model.setProperty("/subjectName", "");
 
+      this._model.setProperty("/schemaUploadUrl", window.apiUrl + "/schema/upload");
+
       // initially, registry integration is disabled in UI - get enabled by querying SchemaApiFeatures
       this._model.setProperty("/registryEnabled", false);
       this.checkRegistryIntegration()
@@ -166,6 +168,10 @@ sap.ui.define([
           name: "X-CSRF-TOKEN",
           value: localStorage.getItem("csrfToken")
         }));
+        oFileUpload.addHeaderParameter(new sap.ui.unified.FileUploaderParameter({
+          name: "JWT",
+          value: localStorage.getItem("jwtToken")
+        }));
         oFileUpload.upload();
       }
     },
@@ -184,12 +190,13 @@ sap.ui.define([
         };
 
         jQuery.ajax({
-          url: "api/schema/registry",
+          url: window.apiUrl + "/schema/registry",
           type: 'POST',
           data: $.param(data),
           contentType: 'application/x-www-form-urlencoded',
           context: this, // becomes the result of "this" in handleRemoteLoad*Complete
           headers: {
+            "JWT": localStorage.getItem("jwtToken"),
             'X-CSRF-TOKEN': localStorage.getItem("csrfToken")
           },
           complete: this.handleRemoteLoadFromSubjectNameComplete
@@ -216,7 +223,7 @@ sap.ui.define([
       };
 
       jQuery.ajax({
-        url: "api/schema/remote",
+        url: window.apiUrl + "/schema/remote",
         type: 'POST',
         data: $.param(data),
         contentType: 'application/x-www-form-urlencoded',
@@ -388,8 +395,12 @@ sap.ui.define([
 
     checkRegistryIntegration: function () {
       jQuery.ajax({
-        url: "api/schema/features",
+        url: window.apiUrl + "/schema/features",
         type: 'GET',
+        headers: {
+          "JWT": localStorage.getItem("jwtToken"),
+          'X-CSRF-TOKEN': localStorage.getItem("csrfToken")
+        },
         context: this,
         complete: this.handleRegistryIntegrationResponse
       });