Aws-POC merge to develop #1632

benedeki · 2021-01-11T01:55:51Z

Upgrade to Atum 3.2.2 // @dk1844: I think it is 3.2.0
Menas Docker Setup
Kept spline 0.3
Restore/Kept HDFS (file) browser in Menas
Restore/Kept Oozie in Menas
Preserving all other changes from AWS-PoC branch

% Conflicts: % dao/pom.xml % data-model/pom.xml % examples/pom.xml % menas/pom.xml % menas/src/main/resources/application.properties.template % migrations-cli/pom.xml % migrations/pom.xml % plugins-api/pom.xml % plugins-builtin/pom.xml % pom.xml % spark-jobs/pom.xml % utils/pom.xml

…2-snapshot Aws poc update from develop 2.12 snapshot

- all directly-hdfs touching stuff disabled (atum, performance measurements, info files, output path checking) # Add menasfargate into hosts sudo nano /etc/hosts # paste 20.0.63.69 menasfargate # save & exit (ctrl+O, ctrl+X) # Running standardization works via: spark-submit --class za.co.absa.enceladus.standardization.StandardizationJob --conf "spark.driver.extraJavaOptions=-Dmenas.rest.uri=http://menasfargate:8080 -Dstandardized.hdfs.path=s3://euw1-ctodatadev-dev-bigdatarnd-s3-poc/enceladusPoc/ao-hdfs-data/stdOutput/standardized-{0}-{1}-{2}-{3}" ~/enceladusPoc/spark-jobs-2.11.0-SNAPSHOT.jar --menas-credentials-file ~/enceladusPoc/menas-credentials.properties --dataset-name dk_test1_emr285 --raw-format json --dataset-version 1 --report-date 2019-11-27 --report-version 1 2> ~/enceladusPoc/stderr.txt

…ut, s3 conf output) 0- all directly-hdfs touching stuff disabled (atum, performance measurements, info files, output path checking) # Add menasfargate into hosts sudo nano /etc/hosts # paste 20.0.63.69 menasfargate # save & exit (ctrl+O, ctrl+X) # Running conformance works via: spark-submit --class za.co.absa.enceladus.conformance.DynamicConformanceJob --conf "spark.driver.extraJavaOptions=-Dmenas.rest.uri=http://menasfargate:8080 -Dstandardized.hdfs.path=s3://euw1-ctodatadev-dev-bigdatarnd-s3-poc/enceladusPoc/ao-hdfs-data/stdOutput/standardized-{0}-{1}-{2}-{3}" ~/enceladusPoc/spark-jobs-2.11.0-SNAPSHOT.jar --menas-credentials-file ~/enceladusPoc/menas-credentials.properties --dataset-name dk_test1_emr285 --dataset-version 1 --report-date 2019-11-27 --report-version 1 2> ~/enceladusPoc/conf-log.txt

% Conflicts: % pom.xml

Feature/1416 S3 input/output for Enceladus on EMR (S2.4, H2.8.5)

This is a temporary solution. We currently experiment with many forms of URLs, and having a regex there now slows us down.

#1526 * FsUtils divided into LocalFsUtils & HdfsUtils * PathConfigSuite update * S3FsUtils with tail-recursive pagination accumulation - now generic with optional short-circuit breakOut TestRunnerJob updated to manually cover the cases - should serve as a basis for tests * HdfsUtils replace by trait DistributedFsUtils (except for MenasCredentials loading & nonSplittable splitting) * using final version of s3-powered Atum (3.0.0) * mockito-update version update, scalatest version update * S3FsUtilsSuite: exists, read, sizeDir(hidden, non-hidden, reucursive), non-splittable (simple, recursive with breakOut), delete (recursive), version find (simple - empty, recursive) * explicit stubbing fix for hyperdrive

* s3 using hadoop fs api * s3 sdk usage removed (pom, classes, tests) * atum final version 3.1.0 used * readStandardizationInputData(... path: String)(implicit ... fs: FileSystem) -> readStandardizationInputData(input: PathWithFs)

* #1554 Tomcat with TLS container * #1554 Added envoy config + enabling running unencrypted container

1499 Make Spline in Menas work for AWS

…e commons library version. Test-ran on EMR. (#1619)

* Tests pass * Depends on ATUM 3.1.1-SNAPSHOT (the bugfix for AbsaOSS/atum#48)

* put back Oozie * downgraded Spline

Conflicts addressed: # spark-jobs/src/test/scala/za/co/absa/enceladus/conformance/datasource/DatasourceSuite.scala # spark-jobs/src/test/scala/za/co/absa/enceladus/conformance/interpreter/rules/RuleOptimizationSuite.scala # spark-jobs/src/test/scala/za/co/absa/enceladus/conformance/interpreter/rules/RulesSuite.scala # spark-jobs/src/test/scala/za/co/absa/enceladus/conformance/interpreter/rules/mapping/JoinMappingRuleInterpreterSuite.scala

AdrianOlosutean · 2021-01-12T08:50:49Z

spark-jobs/src/main/scala/za/co/absa/enceladus/common/CommonJobExecution.scala

    jobConfig.reportVersion match {
      case Some(version) => version
      case None =>
+
+        // publishFs for this specific feature (needed for missing reportVersion until reusable


Is this comment needed anymore?

I am not even sure what the comment means. 🤔 @dk1844 ?

I believe the comment is needed (or at least that it is not meaningless). Maybe it should be reworded though if it is unclear. The gist of the meaning is this:

Everywhere else, one would want to reuse existing "Publish FS" that will reside in pathCfg.publish.fileSystem in this situation (to get an instance of HadoopFsUtils bound to this FS). However, here, you cannot do it, because pathCfg object is not yet available at this point. So, you the would-be-publishFs is here obtained temporarily from the dataset.hdfsPublishPath.

That's all. The comment just meant to explain why seemingly the call of dataset.hdfsPublishPath is pointlessly repeated here and also later in the factory method of PathConfig. It just could not be reused reasonably.

My suggestion for the comment here is:

// Since "pathConfig.publish.fileSystem" is not available at this point yet, a temporary publish-FS is used here instead

This should explain the intention quite a bit and not be too complicated.

AdrianOlosutean · 2021-01-12T09:27:54Z

utils/src/main/scala/za/co/absa/enceladus/utils/fs/HadoopFsUtils.scala

    val in = fs.open(new Path(hdfsPath))
    val content = Array.fill(in.available())(0.toByte)
    in.readFully(content)
    val tmpFile = File.createTempFile("enceladusFSUtils", "hdfsFileToLocalTemp")
    tmpFile.deleteOnExit()
    FileUtils.writeByteArrayToFile(tmpFile, content)
    tmpFile.getAbsolutePath
+
+    // why not use


Can this be removed?

Seems so. @dk1844 can you confirm, please?

The comment still makes sense to me. Why in this case we do not use the simpler call as outlined in the comment:

fs.copyToLocalFile(false, new Path(hdfsPath), new Path("someLocalName"), true)

AdrianOlosutean · 2021-01-13T14:51:06Z

pom.xml

+        <scala.version>2.11.12</scala.version>
+        <scalatest.maven.version>2.0.0</scalatest.maven.version>
+        <scalatest.version>3.2.2</scalatest.version>
+        <scopt.version>4.0.0-RC2</scopt.version>


Can scopt be updated to the 4.0.0 release?

dk1844

(just read through, just small non-blocking comments)

menas/ui/css/style.css

dk1844 · 2021-01-13T15:26:38Z

menas/ui/css/style.css

+  margin-top: 1rem !important;
+}
+
+h5[id$=datasetDetailView--scheduleTimingTitle] {


Possible, but quite ugly. Couldn't the element just have a custom CSS class instead of relying on the generated id?

I agree it's not nice. But I don't want to add improvements (not required by the merge) into the PR. Please consider fixing it, and creating a dedicated PR for it.

menas/ui/service/RunService.js

menas/pom.xml

README.md

menas/ui/components/dataset/monitoring/barChartPopover.fragment.xml

scripts/bash/enceladus_env.template.sh

scripts/bash/run_enceladus.sh

* Addressed issues identified by reviewers

benedeki

Sorry for this PR being so big. 😊

README.md

menas/pom.xml

menas/ui/components/dataset/monitoring/barChartPopover.fragment.xml

menas/ui/css/style.css

benedeki · 2021-01-19T23:54:03Z

menas/ui/css/style.css

+  margin-top: 1rem !important;
+}
+
+h5[id$=datasetDetailView--scheduleTimingTitle] {


I agree it's not nice. But I don't want to add improvements (not required by the merge) into the PR. Please consider fixing it, and creating a dedicated PR for it.

menas/ui/service/RunService.js

Conflicts addressed: # pom.xml # spark-jobs/src/test/scala/za/co/absa/enceladus/conformance/interpreter/InterpreterSuite.scala # utils/src/test/scala/za/co/absa/enceladus/utils/config/ConfigReaderSuite.scala

menas/ui/css/style.css

HuvarVer

I did some tests

spark jobs

complex standardization
conformance with almost all of the rules
standardization&conformance with mapping table rule

UI

create and update schema
create and update mapping table
create and update dataset and conformance rules

Comparison with data on current Develop branche

Zejnilovic

Code reviewed
No functional changes after Veronika's approve

…eladus into aws-merge-to-develop

sonarqubecloud · 2021-01-29T13:49:36Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
3 Code Smells

No Coverage information
1.1% Duplication

AdrianOlosutean

Code reviewed

Zejnilovic and others added 29 commits July 2, 2020 19:48

1422 and 1423 Remove HDFS and Oozie from Menas

18cb47f

#1422 Fix HDFS location validation

dcde55e

#1424 Add Menas Dockerfile

a817cfe

Merge pull request #1484 from AbsaOSS/aws-poc-update-from-develop-2.1…

ff10529

…2-snapshot Aws poc update from develop 2.12 snapshot

#1416 hadoop-aws 2.8.5 + s3 aws sdk 2.13.65 compiles.

60d2948

ref issue = 1416

ac0785d

related test cases ignored (issue reference added)

a8d53f9

PR updates

85570d7

Merge spline 0.5.3 into aws-poc

7be3afb

Update spline to 0.5.4 for AWS PoC

aa69593

Merge branch 'aws-poc' into feature/1416-aws-emr-poc

0bd704c

% Conflicts: % pom.xml

Merge pull request #1483 from AbsaOSS/feature/1416-aws-emr-poc

0b60b1a

Feature/1416 S3 input/output for Enceladus on EMR (S2.4, H2.8.5)

#1503 Remove HDFS url Validation

0450459

This is a temporary solution. We currently experiment with many forms of URLs, and having a regex there now slows us down.

New dockerfile - smaller image

2b3c39c

1554 Tomcat with TLS in Docker container (#1585)

4598946

* #1554 Tomcat with TLS container * #1554 Added envoy config + enabling running unencrypted container

#1499 Add authentication to /lineage + update spline to 0.5.5

a9efecd

Merge pull request #1606 from AbsaOSS/feature/1499-spline-in-menas-aws

5da8e36

1499 Make Spline in Menas work for AWS

#1618 - fixes failing spline 0.5.5 integration by providing compatibl…

34aefbb

…e commons library version. Test-ran on EMR. (#1619)

Merge branch 'aws-poc' into aws-merge-to-develop

b4debed

WIP fixing merge issues

c6094e1

* Merge compiles

9d39ca0

* Tests pass * Depends on ATUM 3.1.1-SNAPSHOT (the bugfix for AbsaOSS/atum#48)

* put back HDFS browser

9e277b2

* put back Oozie * downgraded Spline

* AWS SDK Exclusion

4689308

* New ATUM version

0d95c42

benedeki requested a review from AdrianOlosutean as a code owner January 11, 2021 01:55

benedeki requested review from DzMakatun, HuvarVer and lokm01 as code owners January 11, 2021 01:55

benedeki self-assigned this Jan 11, 2021

benedeki added the work in progress Work on this item is not yet finished (mainly intended for PRs) label Jan 11, 2021

benedeki added 2 commits January 11, 2021 03:44

* Adding missing files

267a457

benedeki removed the work in progress Work on this item is not yet finished (mainly intended for PRs) label Jan 11, 2021

AdrianOlosutean reviewed Jan 13, 2021

View reviewed changes

dk1844 previously approved these changes Jan 13, 2021

View reviewed changes

Zejnilovic reviewed Jan 19, 2021

View reviewed changes

README.md Outdated Show resolved Hide resolved

menas/ui/components/dataset/monitoring/barChartPopover.fragment.xml Show resolved Hide resolved

scripts/bash/enceladus_env.template.sh Outdated Show resolved Hide resolved

scripts/bash/run_enceladus.sh Outdated Show resolved Hide resolved

1622: Merge of aws-poc to develop brach

8702f88

* Addressed issues identified by reviewers

benedeki dismissed dk1844’s stale review via 8702f88 January 20, 2021 00:59

benedeki commented Jan 20, 2021

View reviewed changes

Merge branch 'develop' into aws-merge-to-develop

7999644

Conflicts addressed: # pom.xml # spark-jobs/src/test/scala/za/co/absa/enceladus/conformance/interpreter/InterpreterSuite.scala # utils/src/test/scala/za/co/absa/enceladus/utils/config/ConfigReaderSuite.scala

Zejnilovic reviewed Jan 20, 2021

View reviewed changes

menas/ui/css/style.css Show resolved Hide resolved

HuvarVer previously approved these changes Jan 22, 2021

View reviewed changes

* comments improvement

2391788

benedeki dismissed HuvarVer’s stale review via 2391788 January 22, 2021 21:28

Zejnilovic and others added 2 commits January 25, 2021 15:21

1434 Add new way of serving properties to Docker

3050cbc

Merge branch 'develop' into aws-merge-to-develop

88bcd9a

Zejnilovic previously approved these changes Jan 29, 2021

View reviewed changes

benedeki added 2 commits January 29, 2021 14:46

* Scopt 4.0.0

d0af27b

Merge branch 'aws-merge-to-develop' of https://github.com/AbsaOSS/enc…

8b0634d

…eladus into aws-merge-to-develop

benedeki dismissed Zejnilovic’s stale review via 8b0634d January 29, 2021 13:46

Zejnilovic approved these changes Jan 29, 2021

View reviewed changes

AdrianOlosutean approved these changes Jan 29, 2021

View reviewed changes

benedeki merged commit 722ff05 into develop Jan 29, 2021

benedeki deleted the aws-merge-to-develop branch January 29, 2021 15:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Aws-POC merge to develop #1632

Aws-POC merge to develop #1632

benedeki commented Jan 11, 2021 •

edited by dk1844

Loading

AdrianOlosutean Jan 12, 2021

benedeki Jan 20, 2021

dk1844 Jan 21, 2021 •

edited

Loading

dk1844 Jan 22, 2021 •

edited

Loading

AdrianOlosutean Jan 12, 2021

benedeki Jan 20, 2021

dk1844 Jan 21, 2021

AdrianOlosutean Jan 13, 2021

dk1844 left a comment

dk1844 Jan 13, 2021

benedeki Jan 19, 2021

benedeki left a comment

benedeki Jan 19, 2021

HuvarVer left a comment •

edited

Loading

Zejnilovic left a comment

sonarqubecloud bot commented Jan 29, 2021

AdrianOlosutean left a comment

Aws-POC merge to develop #1632

Aws-POC merge to develop #1632

Conversation

benedeki commented Jan 11, 2021 • edited by dk1844 Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dk1844 Jan 21, 2021 • edited Loading

Choose a reason for hiding this comment

dk1844 Jan 22, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dk1844 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benedeki left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HuvarVer left a comment • edited Loading

Choose a reason for hiding this comment

Zejnilovic left a comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Jan 29, 2021

AdrianOlosutean left a comment

Choose a reason for hiding this comment

benedeki commented Jan 11, 2021 •

edited by dk1844

Loading

dk1844 Jan 21, 2021 •

edited

Loading

dk1844 Jan 22, 2021 •

edited

Loading

HuvarVer left a comment •

edited

Loading