Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Separate Menas into two Tomcat deployable applications #1612

Closed
AdrianOlosutean opened this issue Dec 3, 2020 · 0 comments
Closed

Separate Menas into two Tomcat deployable applications #1612

AdrianOlosutean opened this issue Dec 3, 2020 · 0 comments
Assignees
@AdrianOlosutean
Labels
Menas Menas component affected

Comments

@AdrianOlosutean
Copy link
Contributor

No description provided.

@AdrianOlosutean AdrianOlosutean self-assigned this Dec 3, 2020
@AdrianOlosutean AdrianOlosutean added the Menas Menas component affected label Dec 3, 2020
AdrianOlosutean pushed a commit that referenced this issue Dec 16, 2020
@Adrian-Olosutean
#1612 Separation start
94586ad
@AdrianOlosutean AdrianOlosutean mentioned this issue Dec 16, 2020
Merged
AdrianOlosutean pushed a commit that referenced this issue Dec 17, 2020
@Adrian-Olosutean
#1612 Updated DAO for spark-jobs
3f7e513
AdrianOlosutean pushed a commit that referenced this issue Dec 18, 2020
@Adrian-Olosutean
#1612 Fixed spline integration and schema, removed redundant code
17b19e1
AdrianOlosutean pushed a commit that referenced this issue Dec 18, 2020
@Adrian-Olosutean
#1612 Fixed tests, removed unused dependency
8cb6521
AdrianOlosutean pushed a commit that referenced this issue Dec 18, 2020
@Adrian-Olosutean
#1612 Added back dependency
f90ff2d
AdrianOlosutean pushed a commit that referenced this issue Jan 5, 2021
@Adrian-Olosutean
#1612 Removed Spring from menas-web, enabled building war and static …
d233991 
…resources. Removed version subpath in menas-web + added theme dependencies in repo
AdrianOlosutean pushed a commit that referenced this issue Jan 6, 2021
@Adrian-Olosutean
#1612 Cookies + updated lineage
5b9ebcd
AdrianOlosutean pushed a commit that referenced this issue Jan 8, 2021
@Adrian-Olosutean
#1612 Included HDFSFolder + missing Oozie parts
a6458ba
AdrianOlosutean pushed a commit that referenced this issue Jan 13, 2021
@Adrian-Olosutean
#1612 menas-web on nginx container and passing API_URL
86c3e00
AdrianOlosutean pushed a commit that referenced this issue Jan 20, 2021
@Adrian-Olosutean
#1612 Working TLS on nginx, resources not included in code
6dec360
AdrianOlosutean pushed a commit that referenced this issue Jan 26, 2021
@Adrian-Olosutean
#1612 Building using ui5 + reused /api route
34807fe
AdrianOlosutean pushed a commit that referenced this issue Jan 26, 2021
@Adrian-Olosutean
#1612 Project version
3226a7f
AdrianOlosutean pushed a commit that referenced this issue Feb 1, 2021
@Adrian-Olosutean
#1612 Merges
42b0e4f
AdrianOlosutean pushed a commit that referenced this issue Feb 1, 2021
@Adrian-Olosutean
#1612 pom parent version
5781874
AdrianOlosutean pushed a commit that referenced this issue Feb 10, 2021
@Adrian-Olosutean
#1612 other fixes
2140ce5
AdrianOlosutean pushed a commit that referenced this issue Feb 10, 2021
@Adrian-Olosutean
#1612 added pom package.json version sync
5aa0f13
AdrianOlosutean pushed a commit that referenced this issue Feb 10, 2021
@Adrian-Olosutean
#1612 newline
a52af47
AdrianOlosutean pushed a commit that referenced this issue Feb 11, 2021
@Adrian-Olosutean
#1612 fix version sync + cleaning dist
e1280b9
AdrianOlosutean pushed a commit that referenced this issue Mar 19, 2021
@Adrian-Olosutean
#1612 Updated npm project versions + mvn plugin
7f86526
AdrianOlosutean added a commit that referenced this issue Apr 12, 2021
@AdrianOlosutean
#1612 license_check.yml
2771077
dk1844 added a commit that referenced this issue Apr 12, 2021
@dk1844
#1612 licence check fix
a06d01c
AdrianOlosutean added a commit that referenced this issue Apr 12, 2021
@AdrianOlosutean @Zejnilovic
@dk1844 @lokm01 @benedeki
#1612 Separating Menas UI and API (#1620)
0d265e5 
* 1422 and 1423 Remove HDFS and Oozie from Menas

* #1422 Fix HDFS location validation

* #1424 Add Menas Dockerfile

* #1416 hadoop-aws 2.8.5 + s3 aws sdk 2.13.65 compiles.

* #1416 - enceladus on S3:

 - all directly-hdfs touching stuff disabled (atum, performance measurements, info files, output path checking)

# Add menasfargate into hosts
sudo nano /etc/hosts
# paste
20.0.63.69 menasfargate
# save & exit (ctrl+O, ctrl+X)

# Running standardization works via:
spark-submit --class za.co.absa.enceladus.standardization.StandardizationJob --conf "spark.driver.extraJavaOptions=-Dmenas.rest.uri=http://menasfargate:8080 -Dstandardized.hdfs.path=s3://euw1-ctodatadev-dev-bigdatarnd-s3-poc/enceladusPoc/ao-hdfs-data/stdOutput/standardized-{0}-{1}-{2}-{3}" ~/enceladusPoc/spark-jobs-2.11.0-SNAPSHOT.jar --menas-credentials-file ~/enceladusPoc/menas-credentials.properties --dataset-name dk_test1_emr285 --raw-format json --dataset-version 1 --report-date 2019-11-27 --report-version 1 2> ~/enceladusPoc/stderr.txt

* #1416 - enceladus on S3 - (crude) conformance works on s3 (s3 std input, s3 conf output)

 0- all directly-hdfs touching stuff disabled (atum, performance measurements, info files, output path checking)

# Add menasfargate into hosts
sudo nano /etc/hosts
# paste
20.0.63.69 menasfargate
# save & exit (ctrl+O, ctrl+X)

# Running conformance works via:
spark-submit --class za.co.absa.enceladus.conformance.DynamicConformanceJob --conf "spark.driver.extraJavaOptions=-Dmenas.rest.uri=http://menasfargate:8080 -Dstandardized.hdfs.path=s3://euw1-ctodatadev-dev-bigdatarnd-s3-poc/enceladusPoc/ao-hdfs-data/stdOutput/standardized-{0}-{1}-{2}-{3}" ~/enceladusPoc/spark-jobs-2.11.0-SNAPSHOT.jar --menas-credentials-file ~/enceladusPoc/menas-credentials.properties --dataset-name dk_test1_emr285 --dataset-version 1 --report-date 2019-11-27 --report-version 1 2> ~/enceladusPoc/conf-log.txt

* ref issue = 1416

* related test cases ignored (issue reference added)

* PR updates

* Merge spline 0.5.3 into aws-poc

* Update spline to 0.5.4 for AWS PoC

* #1503 Remove HDFS url Validation

This is a temporary solution. We currently experiment with
many forms of URLs, and having a regex there now slows us down.

* New dockerfile - smaller image

* s3 persistence (atum, sdk fs usage, ...) (#1526)

#1526 
* FsUtils divided into LocalFsUtils & HdfsUtils
* PathConfigSuite update
* S3FsUtils with tail-recursive pagination accumulation - now generic with optional short-circuit breakOut
TestRunnerJob updated to manually cover the cases - should serve as a basis for tests
* HdfsUtils replace by trait DistributedFsUtils (except for MenasCredentials loading & nonSplittable splitting)
* using final version of s3-powered Atum (3.0.0)
* mockito-update version update, scalatest version update
* S3FsUtilsSuite: exists, read, sizeDir(hidden, non-hidden, reucursive), non-splittable (simple, recursive with breakOut), delete (recursive), version find (simple - empty, recursive)
* explicit stubbing fix for hyperdrive

* Feature/1556 file access PoC using Hadoop FS API (#1586)

* s3 using hadoop fs api
* s3 sdk usage removed (pom, classes, tests)
* atum final version 3.1.0 used
* readStandardizationInputData(... path: String)(implicit ... fs: FileSystem) -> readStandardizationInputData(input: PathWithFs)

* 1554 Tomcat with TLS in Docker container (#1585)

* #1554 Tomcat with TLS container

* #1554 Added envoy config + enabling running unencrypted container

* #1499 Add authentication to /lineage + update spline to 0.5.5

* #1618 - fixes failing spline 0.5.5 integration by providing compatible commons library version. Test-ran on EMR. (#1619)

* #1612 Separation start

* #1612 Updated DAO for spark-jobs

* #1612 Fixed spline integration and schema, removed redundant code

* #1612 Fixed tests, removed unused dependency

* #1612 Added back dependency

* WIP fixing merge issues

* * Merge compiles
* Tests pass
* Depends on ATUM 3.1.1-SNAPSHOT (the bugfix for AbsaOSS/atum#48)

* #1612 Removed Spring from menas-web, enabled building war and static resources. Removed version subpath in menas-web + added theme dependencies in repo

* #1612 Cookies + updated lineage

* * put back HDFS browser
* put back Oozie
* downgraded Spline

* * AWS SDK Exclusion

* #1612 Included HDFSFolder + missing Oozie parts

* * New ATUM version

* * Adding missing files

* #1612 menas-web on nginx container and passing API_URL

* #1612 Working TLS on nginx, resources not included in code

* 1622: Merge of aws-poc to develop brach
* Addressed issues identified by reviewers

* * comments improvement

* 1434 Add new way of serving properties to Docker

* #1612 Building using ui5 + reused /api route

* #1612 Project version

* #713 Add favicon

* #1612 Merges

* #1612 pom parent version

* #1648 Fix war deployment + adding back spline to menas

* #1612 other fixes

* #1612 added pom package.json version sync

* #1612 newline

* #1612 fix version sync + cleaning dist

* 1648 merge to develop

* 1648 merge fix

* 1648 Fixes schema upload

* 1648 Fixes schema registry request

* 1648 pom version

* 1612 add docker build

* #601 Swagger 2 PoC

* #601 Swagger 2 PoC

* #601 Swagger 2 PoC

* #1648 Updating menas-web to 3.0

* #1612 Updated npm project versions + mvn plugin

* #1612 license_check.yml

* #1612 licence check fix

Co-authored-by: Saša Zejnilović <zejnils@gmail.com>
Co-authored-by: Daniel Kavan <dk1844@gmail.com>
Co-authored-by: Jan Scherbaum <kmoj02@gmail.com>
Co-authored-by: David Benedeki <benedeki@volny.cz>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@AdrianOlosutean AdrianOlosutean

Labels
Menas Menas component affected
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@AdrianOlosutean