1732 Spline 0.6 integration #1739

AdrianOlosutean · 2021-03-22T09:25:19Z

* #1126: Spline 0.4.x in Oozie * new setting for Spline used by Oozie * changed Oozie template * optional spline.mode setting support * applied some code hints

#1110: Make Standardization and Conformance use Spline 0.4.x * Change Spline version * Make **Standardization** and **Confluence** depend and init the new Spline module instead of the old ones * Replacing Spline properties in templates * Using a different JSON library (instead of the one transitionally ingested from old Spline) * Application properties templates adjutsted * Helper scripts updated * README.md updated (addressing also changes in Menas and Oozie from other PR) * Missing copyright headers Co-authored-by: Georgi Chochov <g.chochov@gmail.com>

* Server-side api endpoints to provide new configuration * Server-side controller to serve the Spline WebJar * Client-side changes to gather the lineage id and display the lineage new UI * Added dependency to include the webjar * Header to allow iframe display `<frame-options policy="SAMEORIGIN"/>` * fixed `style.css`

* Rebasing to current `Develop`

* Unused import removed

…ncies * Version names sorted alphabetically

# Conflicts (resolved): # pom.xml # scripts/bash/run_enceladus.sh # spark-jobs/pom.xml

% Conflicts: % dao/pom.xml % data-model/pom.xml % examples/pom.xml % menas/pom.xml % menas/src/main/resources/application.properties.template % migrations-cli/pom.xml % migrations/pom.xml % plugins-api/pom.xml % plugins-builtin/pom.xml % pom.xml % spark-jobs/pom.xml % utils/pom.xml

…2-snapshot Aws poc update from develop 2.12 snapshot

- all directly-hdfs touching stuff disabled (atum, performance measurements, info files, output path checking) # Add menasfargate into hosts sudo nano /etc/hosts # paste 20.0.63.69 menasfargate # save & exit (ctrl+O, ctrl+X) # Running standardization works via: spark-submit --class za.co.absa.enceladus.standardization.StandardizationJob --conf "spark.driver.extraJavaOptions=-Dmenas.rest.uri=http://menasfargate:8080 -Dstandardized.hdfs.path=s3://euw1-ctodatadev-dev-bigdatarnd-s3-poc/enceladusPoc/ao-hdfs-data/stdOutput/standardized-{0}-{1}-{2}-{3}" ~/enceladusPoc/spark-jobs-2.11.0-SNAPSHOT.jar --menas-credentials-file ~/enceladusPoc/menas-credentials.properties --dataset-name dk_test1_emr285 --raw-format json --dataset-version 1 --report-date 2019-11-27 --report-version 1 2> ~/enceladusPoc/stderr.txt

…ut, s3 conf output) 0- all directly-hdfs touching stuff disabled (atum, performance measurements, info files, output path checking) # Add menasfargate into hosts sudo nano /etc/hosts # paste 20.0.63.69 menasfargate # save & exit (ctrl+O, ctrl+X) # Running conformance works via: spark-submit --class za.co.absa.enceladus.conformance.DynamicConformanceJob --conf "spark.driver.extraJavaOptions=-Dmenas.rest.uri=http://menasfargate:8080 -Dstandardized.hdfs.path=s3://euw1-ctodatadev-dev-bigdatarnd-s3-poc/enceladusPoc/ao-hdfs-data/stdOutput/standardized-{0}-{1}-{2}-{3}" ~/enceladusPoc/spark-jobs-2.11.0-SNAPSHOT.jar --menas-credentials-file ~/enceladusPoc/menas-credentials.properties --dataset-name dk_test1_emr285 --dataset-version 1 --report-date 2019-11-27 --report-version 1 2> ~/enceladusPoc/conf-log.txt

% Conflicts: % pom.xml

Feature/1416 S3 input/output for Enceladus on EMR (S2.4, H2.8.5)

This is a temporary solution. We currently experiment with many forms of URLs, and having a regex there now slows us down.

…32-spline-06-integration # Conflicts: # menas-web/Dockerfile # menas-web/nginx.conf # menas-web/start_menas_web.sh # menas-web/ui/index.html # menas-web/ui/package.json # menas-web/ui/service/RestDAO.js # menas-web/ui/service/RunRestDAO.js # menas-web/ui/ui5.yaml # menas/src/main/resources/start_menas.sh # menas/src/main/scala/za/co/absa/enceladus/menas/WebSecurityConfig.scala

benedeki · 2021-04-17T09:30:22Z

Just curious why you included the Swagger in this PR? 😉

AdrianOlosutean · 2021-04-19T07:54:44Z

Just curious why you included the Swagger in this PR? 😉

I considered it's quite useful to add it after separating Menas. If you think it's better, I will put the changes in a separate PR

benedeki

Just curious why you included the Swagger in this PR? 😉

I considered it's quite useful to add it after separating Menas. If you think it's better, I will put the changes in a separate PR

It is definitely useful, no doubt. But I see no crossover with the Spline integration. So having it in one PR - particularly when there already was a PR (maybe not on separated Menas) - seems contra productive, just increases the size of the PR.
Furthermore Spline is dragging behind a little. Swagger could be already merged.... Perhpas it would be worth to separate into its own PR. 🤔

benedeki · 2021-04-21T12:28:49Z

README.md

@@ -119,7 +119,7 @@ password=changeme
 --deploy-mode <client/cluster> \
 --driver-cores <num> \
 --driver-memory <num>G \
--conf "spark.driver.extraJavaOptions=-Dmenas.rest.uri=<menas_api_uri:port> -Dstandardized.hdfs.path=<path_for_standardized_output>-{0}-{1}-{2}-{3} -Dspline.mongodb.url=<mongo_url_for_spline> -Dspline.mongodb.name=<spline_database_name> -Dhdp.version=<hadoop_version>" \
+--conf "spark.driver.extraJavaOptions=-Dmenas.rest.uri=<menas_api_uri:port> -Dstandardized.hdfs.path=<path_for_standardized_output>-{0}-{1}-{2}-{3} -Dspline.producer.url=<url_for_spline_consumer> -Dhdp.version=<hadoop_version>" \


In an effort to decouple Enceladus and Spline, I wouldn't even mention this option here. If in application.conf it's good enough, and can be overridden as any other parameter.

spark-jobs/src/main/resources/spline.properties.template

menas-web/ui/index.html

menas-web/ui/package.json

menas-web/ui/service/RestDAO.js

benedeki · 2021-04-21T13:32:33Z

menas/src/main/scala/za/co/absa/enceladus/menas/HDFSConfig.scala

@@ -28,7 +28,7 @@ import org.springframework.context.annotation.{Bean, Configuration}
 class HDFSConfig @Autowired() (spark: SparkSession) {
  private val logger = LoggerFactory.getLogger(this.getClass)

-  @Value("${menas.hadoop.auth.method}")
+  @Value("${menas.hadoop.auth.method:}")


Why the colon?

It used to fail before because of this, but we solved it another way. Anyway, it seems to be the only one without a colon here

Let's not forget what the colon means - in general, it changes the meaning of the property ${"my.property.key"} (without default value - so failing to find it in the config should cause a failure) to ${"my.property.key:defaultValueForThatKey"} (where failing to find the key in config just fallbacks to the default - and the nothing after the colon means default empty string, but having default provided nonetheless)

Most probably, you all knew all this, just stating it so we "just don't add" colons because it is present elsewhere, but we consider the true meaning - and here that it should be whether it is reasonable to provide default empty string when the key is not found in properties.

benedeki · 2021-04-21T15:05:20Z

menas/src/main/resources/application.properties.template

@@ -130,8 +126,11 @@ menas.oozie.mavenSparkJobsJarLocation=/za/co/absa/enceladus/spark-jobs/@project.
 #Menas URL for submitted std and conf jobs
 menas.oozie.menasApiURL=http://menasHostname:8080/menas/api

-#Mongo address for spline for the submitted jobs
-menas.oozie.splineMongoURL=mongodb://localhost:27017
+#The URL address of Spline for the submitted jobs


I would get rid of it Spline reference from Oozie code completely. Let's take if from application.conf.

I also thought of this, but wouldn't it break Oozie integration?

menas/src/main/scala/za/co/absa/enceladus/menas/repositories/OozieRepository.scala

scripts/bash/enceladus_env.template.sh

scripts/bash/run_enceladus.sh

…on' into feature/1732-spline-06-integration

…32-spline-06-integration # Conflicts: # README.md # rest-api/src/main/scala/za/co/absa/enceladus/rest_api/WebSecurityConfig.scala

…32-spline-06-integration # Conflicts: # pom.xml # spark-jobs/pom.xml # spark-jobs/src/main/resources/spline.properties.template

…32-spline-06-integration

…on' into feature/1732-spline-06-integration # Conflicts: # rest-api/src/main/scala/za/co/absa/enceladus/rest_api/WebSecurityConfig.scala

sonarcloud · 2021-06-01T11:42:46Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

dk1844

LGTM (read the code and test-ran on EKS/Fargate with @AdrianOlosutean )

dk1844 · 2021-06-02T08:58:17Z

rest-api/src/main/scala/za/co/absa/enceladus/rest_api/HDFSConfig.scala

@@ -28,7 +28,7 @@ import org.springframework.context.annotation.{Bean, Configuration}
 class HDFSConfig @Autowired() (spark: SparkSession) {
  private val logger = LoggerFactory.getLogger(this.getClass)

-  @Value("${menas.hadoop.auth.method}")
+  @Value("${menas.hadoop.auth.method:}")


Why is the the empty string a desired default value of this field when all properties files in the project use menas.hadoop.auth.method=default. If we need the default value here, shouldn't it rather be default, then?

Suggested change

@Value("${menas.hadoop.auth.method:}")

@Value("${menas.hadoop.auth.method:default}")

dk1844

LGTM (read the code and test-ran on EKS/Fargate with @AdrianOlosutean )

GeorgiChochov and others added 30 commits March 19, 2020 23:02

Update versions for release v2.0.0

870384a

#1090 Change groupId to za.co.absa.enceladus

228eb4f

#1126: Spline 0.4.x in Oozie (#1127)

f925ba3

* #1126: Spline 0.4.x in Oozie * new setting for Spline used by Oozie * changed Oozie template * optional spline.mode setting support * applied some code hints

#735: Integrate Spline 0.4.x

824d9cc

* Rebasing to current `Develop`

#735: Integrate Spline 0.4.x

6c0b1b2

* Unused import removed

Spline 0.4.2

2d10c5f

Merge branch 'develop' into feature/735-integrate-spline-0-4-x

35806d6

Upgrade to Spline 0.5.x

106154d

Menas test application.properties comment

9f15b2f

* Added named versions to main pom.xml for Spline and other depende…

cee0c49

…ncies * Version names sorted alphabetically

Merge branch 'develop' into feature/735-integrate-spline-0-4-x

100c99f

# Conflicts (resolved): # pom.xml # scripts/bash/run_enceladus.sh # spark-jobs/pom.xml

Update to Spline libs 0.5.3

7277669

1422 and 1423 Remove HDFS and Oozie from Menas

18cb47f

#1422 Fix HDFS location validation

dcde55e

#1424 Add Menas Dockerfile

a817cfe

Merge pull request #1484 from AbsaOSS/aws-poc-update-from-develop-2.1…

ff10529

…2-snapshot Aws poc update from develop 2.12 snapshot

#1416 hadoop-aws 2.8.5 + s3 aws sdk 2.13.65 compiles.

60d2948

ref issue = 1416

ac0785d

related test cases ignored (issue reference added)

a8d53f9

PR updates

85570d7

Merge spline 0.5.3 into aws-poc

7be3afb

Update spline to 0.5.4 for AWS PoC

aa69593

Merge branch 'aws-poc' into feature/1416-aws-emr-poc

0bd704c

% Conflicts: % pom.xml

Merge pull request #1483 from AbsaOSS/feature/1416-aws-emr-poc

0b60b1a

Feature/1416 S3 input/output for Enceladus on EMR (S2.4, H2.8.5)

#1503 Remove HDFS url Validation

0450459

This is a temporary solution. We currently experiment with many forms of URLs, and having a regex there now slows us down.

Zejnilovic mentioned this pull request Apr 12, 2021

735: integrate spline 0.5.x #1175

Closed

Base automatically changed from feature/1612-separate-menas-into-2-tomcat-apps to develop-ver-3.0 April 12, 2021 19:13

AdrianOlosutean added 2 commits April 13, 2021 10:31

#1732 merge fixes

6a030b8

Merge branch 'develop-ver-3.0' into feature/1732-spline-06-integration

8587196

benedeki reviewed Apr 21, 2021

View reviewed changes

AdrianOlosutean added 3 commits April 21, 2021 19:07

#1732 Feedback, merge comments + removed swagger

c13555b

Merge remote-tracking branch 'origin/feature/1732-spline-06-integrati…

f3a8979

…on' into feature/1732-spline-06-integration

#1732 fix build issues

d7e8dac

AdrianOlosutean mentioned this pull request Apr 23, 2021

Feautre/1733 lineage dumper - 2nd edition #1766

Merged

AdrianOlosutean added 6 commits May 7, 2021 12:44

Merge branch 'develop-ver-3.0' into feature/1732-spline-06-integration

9d0d5a9

#1770 Rename menas-web to menas

6f5066f

Merge branch 'feature/1770-rename-menas-web-to-menas' into feature/17…

0199415

…32-spline-06-integration # Conflicts: # README.md # rest-api/src/main/scala/za/co/absa/enceladus/rest_api/WebSecurityConfig.scala

Merge remote-tracking branch 'origin/develop-ver-3.0' into feature/17…

faa7a1f

…32-spline-06-integration # Conflicts: # pom.xml # spark-jobs/pom.xml # spark-jobs/src/main/resources/spline.properties.template

Merge remote-tracking branch 'origin/develop-ver-3.0' into feature/17…

1923529

…32-spline-06-integration

Merge remote-tracking branch 'origin/feature/1732-spline-06-integrati…

2c99bdb

…on' into feature/1732-spline-06-integration # Conflicts: # rest-api/src/main/scala/za/co/absa/enceladus/rest_api/WebSecurityConfig.scala

AdrianOlosutean marked this pull request as ready for review May 27, 2021 08:06

AdrianOlosutean requested review from DzMakatun, HuvarVer and lokm01 as code owners May 27, 2021 08:06

1732 update build lineage url

b8e09cd

dk1844 added the PR:tested Only for PR - PR was tested by a tester (person) label Jun 1, 2021

Merge branch 'develop-ver-3.0' into feature/1732-spline-06-integration

dedb0ae

dk1844 approved these changes Jun 2, 2021

View reviewed changes

AdrianOlosutean merged commit 7f2d6e5 into develop-ver-3.0 Jun 2, 2021

AdrianOlosutean deleted the feature/1732-spline-06-integration branch June 2, 2021 10:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

1732 Spline 0.6 integration #1739

1732 Spline 0.6 integration #1739

AdrianOlosutean commented Mar 22, 2021 •

edited

Loading

benedeki commented Apr 17, 2021

AdrianOlosutean commented Apr 19, 2021

benedeki left a comment

benedeki Apr 21, 2021

benedeki Apr 21, 2021

AdrianOlosutean Apr 21, 2021

dk1844 Apr 26, 2021 •

edited

Loading

benedeki Apr 21, 2021

AdrianOlosutean Apr 21, 2021

sonarcloud bot commented Jun 1, 2021

dk1844 left a comment

dk1844 Jun 2, 2021

dk1844 left a comment

	@Value("${menas.hadoop.auth.method:}")
	@Value("${menas.hadoop.auth.method:default}")

1732 Spline 0.6 integration #1739

1732 Spline 0.6 integration #1739

Conversation

AdrianOlosutean commented Mar 22, 2021 • edited Loading

benedeki commented Apr 17, 2021

AdrianOlosutean commented Apr 19, 2021

benedeki left a comment

Choose a reason for hiding this comment

benedeki Apr 21, 2021

Choose a reason for hiding this comment

benedeki Apr 21, 2021

Choose a reason for hiding this comment

AdrianOlosutean Apr 21, 2021

Choose a reason for hiding this comment

dk1844 Apr 26, 2021 • edited Loading

Choose a reason for hiding this comment

benedeki Apr 21, 2021

Choose a reason for hiding this comment

AdrianOlosutean Apr 21, 2021

Choose a reason for hiding this comment

sonarcloud bot commented Jun 1, 2021

dk1844 left a comment

Choose a reason for hiding this comment

dk1844 Jun 2, 2021

Choose a reason for hiding this comment

dk1844 left a comment

Choose a reason for hiding this comment

AdrianOlosutean commented Mar 22, 2021 •

edited

Loading

dk1844 Apr 26, 2021 •

edited

Loading