Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot specify S3 path in Menas Dataset #1503

Closed
AdrianOlosutean opened this issue Aug 27, 2020 · 0 comments
Closed

Cannot specify S3 path in Menas Dataset #1503

AdrianOlosutean opened this issue Aug 27, 2020 · 0 comments
Labels
bug Something isn't working priority: undecided Undecided priority to be assigned after discussion
Milestone
Enceladus in clou...

Comments

@AdrianOlosutean
Copy link
Contributor

Describe the bug

Menas rejects specifying S3 paths for the raw and publish fields in a dataset.

Expected behaviour

Accept the S3 paths when specified
@AdrianOlosutean AdrianOlosutean added bug Something isn't working priority: undecided Undecided priority to be assigned after discussion labels Aug 27, 2020
@Zejnilovic Zejnilovic added this to the Enceladus in cloud- wave 2 milestone Aug 27, 2020
Zejnilovic added a commit that referenced this issue Aug 27, 2020
@Zejnilovic
#1503 Remove HDFS url Validation
2704d68 
This is a temporary solution. We currently experiment with
many forms of URLs, and having a regex there now slows us down.
@Zejnilovic Zejnilovic mentioned this issue Aug 27, 2020
Merged
Zejnilovic added a commit that referenced this issue Aug 27, 2020
@Zejnilovic
#1503 Remove HDFS url Validation
0450459 
This is a temporary solution. We currently experiment with
many forms of URLs, and having a regex there now slows us down.
benedeki added a commit that referenced this issue Jan 29, 2021
@benedeki @Zejnilovic
@dk1844 @AdrianOlosutean @Adrian-Olosutean @lokm01
Aws-POC merge to develop (#1632)
722ff05 
#1422 and 1423 Remove HDFS and Oozie from Menas

#1422 Fix HDFS location validation

#1424 Add Menas Dockerfile

#1416 hadoop-aws 2.8.5 + s3 aws sdk 2.13.65 compiles.

#1416 - enceladus on S3:
* - all directly-hdfs touching stuff disabled (atum, performance measurements, info files, output path checking)
# Add menasfargate into hosts
# paste
# save & exit (ctrl+O, ctrl+X)

#1416 - enceladus on S3 - (crude) conformance works on s3 (s3 std input, s3 conf output)
* Merge spline 0.5.3 into aws-poc
* Update spline to 0.5.4 for AWS PoC

#1503 Remove HDFS url Validation
* New dockerfile - smaller image
* s3 persistence (atum, sdk fs usage, ...) (#1526)

#1526 
* FsUtils divided into LocalFsUtils & HdfsUtils
* PathConfigSuite update
* S3FsUtils with tail-recursive pagination accumulation - now generic with optional short-circuit breakOut
* TestRunnerJob updated to manually cover the cases - should serve as a basis for tests
* HdfsUtils replace by trait DistributedFsUtils (except for MenasCredentials loading & nonSplittable splitting)
* using final version of s3-powered Atum (3.0.0)
* mockito-update version update, scalatest version update
* S3FsUtilsSuite: exists, read, sizeDir(hidden, non-hidden, reucursive), non-splittable (simple, recursive with breakOut), delete (recursive), version find (simple - empty, recursive)
* explicit stubbing fix for hyperdrive

#1556 file access PoC using Hadoop FS API (#1586)
* s3 using hadoop fs api
* s3 sdk usage removed (pom, classes, tests)
* atum final version 3.1.0 used
* readStandardizationInputData(... path: String)(implicit ... fs: FileSystem) -> readStandardizationInputData(input: PathWithFs)


#1554 Tomcat with TLS container in Docker container

#1554 Added envoy config + enabling running unencrypted container

#1499 Add authentication to /lineage + update spline to 0.5.5

#1618 - fixes failing spline 0.5.5 integration by providing compatible commons library version. Test-ran on EMR. (#1619)

#1434 Add new way of serving properties to Docker

#1622: Merge of aws-poc to develop brach
* put back HDFS browser
* put back Oozie
* downgraded Spline
* Scopt 4.0.0
* AWS SDK Exclusion
* ATUM version 3.2.2

Co-authored-by: Saša Zejnilović <zejnils@gmail.com>
Co-authored-by: Daniel Kavan <dk1844@gmail.com>
Co-authored-by: Adrian Olosutean <adi.olosutean@gmail.com>
Co-authored-by: Adrian Olosutean <adrian.olosutean@absa.africa>
Co-authored-by: Jan Scherbaum <kmoj02@gmail.com>
AdrianOlosutean added a commit that referenced this issue Apr 12, 2021
@AdrianOlosutean @Zejnilovic
@dk1844 @lokm01 @benedeki
#1612 Separating Menas UI and API (#1620)
0d265e5 
* 1422 and 1423 Remove HDFS and Oozie from Menas

* #1422 Fix HDFS location validation

* #1424 Add Menas Dockerfile

* #1416 hadoop-aws 2.8.5 + s3 aws sdk 2.13.65 compiles.

* #1416 - enceladus on S3:

 - all directly-hdfs touching stuff disabled (atum, performance measurements, info files, output path checking)

# Add menasfargate into hosts
sudo nano /etc/hosts
# paste
20.0.63.69 menasfargate
# save & exit (ctrl+O, ctrl+X)

# Running standardization works via:
spark-submit --class za.co.absa.enceladus.standardization.StandardizationJob --conf "spark.driver.extraJavaOptions=-Dmenas.rest.uri=http://menasfargate:8080 -Dstandardized.hdfs.path=s3://euw1-ctodatadev-dev-bigdatarnd-s3-poc/enceladusPoc/ao-hdfs-data/stdOutput/standardized-{0}-{1}-{2}-{3}" ~/enceladusPoc/spark-jobs-2.11.0-SNAPSHOT.jar --menas-credentials-file ~/enceladusPoc/menas-credentials.properties --dataset-name dk_test1_emr285 --raw-format json --dataset-version 1 --report-date 2019-11-27 --report-version 1 2> ~/enceladusPoc/stderr.txt

* #1416 - enceladus on S3 - (crude) conformance works on s3 (s3 std input, s3 conf output)

 0- all directly-hdfs touching stuff disabled (atum, performance measurements, info files, output path checking)

# Add menasfargate into hosts
sudo nano /etc/hosts
# paste
20.0.63.69 menasfargate
# save & exit (ctrl+O, ctrl+X)

# Running conformance works via:
spark-submit --class za.co.absa.enceladus.conformance.DynamicConformanceJob --conf "spark.driver.extraJavaOptions=-Dmenas.rest.uri=http://menasfargate:8080 -Dstandardized.hdfs.path=s3://euw1-ctodatadev-dev-bigdatarnd-s3-poc/enceladusPoc/ao-hdfs-data/stdOutput/standardized-{0}-{1}-{2}-{3}" ~/enceladusPoc/spark-jobs-2.11.0-SNAPSHOT.jar --menas-credentials-file ~/enceladusPoc/menas-credentials.properties --dataset-name dk_test1_emr285 --dataset-version 1 --report-date 2019-11-27 --report-version 1 2> ~/enceladusPoc/conf-log.txt

* ref issue = 1416

* related test cases ignored (issue reference added)

* PR updates

* Merge spline 0.5.3 into aws-poc

* Update spline to 0.5.4 for AWS PoC

* #1503 Remove HDFS url Validation

This is a temporary solution. We currently experiment with
many forms of URLs, and having a regex there now slows us down.

* New dockerfile - smaller image

* s3 persistence (atum, sdk fs usage, ...) (#1526)

#1526 
* FsUtils divided into LocalFsUtils & HdfsUtils
* PathConfigSuite update
* S3FsUtils with tail-recursive pagination accumulation - now generic with optional short-circuit breakOut
TestRunnerJob updated to manually cover the cases - should serve as a basis for tests
* HdfsUtils replace by trait DistributedFsUtils (except for MenasCredentials loading & nonSplittable splitting)
* using final version of s3-powered Atum (3.0.0)
* mockito-update version update, scalatest version update
* S3FsUtilsSuite: exists, read, sizeDir(hidden, non-hidden, reucursive), non-splittable (simple, recursive with breakOut), delete (recursive), version find (simple - empty, recursive)
* explicit stubbing fix for hyperdrive

* Feature/1556 file access PoC using Hadoop FS API (#1586)

* s3 using hadoop fs api
* s3 sdk usage removed (pom, classes, tests)
* atum final version 3.1.0 used
* readStandardizationInputData(... path: String)(implicit ... fs: FileSystem) -> readStandardizationInputData(input: PathWithFs)

* 1554 Tomcat with TLS in Docker container (#1585)

* #1554 Tomcat with TLS container

* #1554 Added envoy config + enabling running unencrypted container

* #1499 Add authentication to /lineage + update spline to 0.5.5

* #1618 - fixes failing spline 0.5.5 integration by providing compatible commons library version. Test-ran on EMR. (#1619)

* #1612 Separation start

* #1612 Updated DAO for spark-jobs

* #1612 Fixed spline integration and schema, removed redundant code

* #1612 Fixed tests, removed unused dependency

* #1612 Added back dependency

* WIP fixing merge issues

* * Merge compiles
* Tests pass
* Depends on ATUM 3.1.1-SNAPSHOT (the bugfix for AbsaOSS/atum#48)

* #1612 Removed Spring from menas-web, enabled building war and static resources. Removed version subpath in menas-web + added theme dependencies in repo

* #1612 Cookies + updated lineage

* * put back HDFS browser
* put back Oozie
* downgraded Spline

* * AWS SDK Exclusion

* #1612 Included HDFSFolder + missing Oozie parts

* * New ATUM version

* * Adding missing files

* #1612 menas-web on nginx container and passing API_URL

* #1612 Working TLS on nginx, resources not included in code

* 1622: Merge of aws-poc to develop brach
* Addressed issues identified by reviewers

* * comments improvement

* 1434 Add new way of serving properties to Docker

* #1612 Building using ui5 + reused /api route

* #1612 Project version

* #713 Add favicon

* #1612 Merges

* #1612 pom parent version

* #1648 Fix war deployment + adding back spline to menas

* #1612 other fixes

* #1612 added pom package.json version sync

* #1612 newline

* #1612 fix version sync + cleaning dist

* 1648 merge to develop

* 1648 merge fix

* 1648 Fixes schema upload

* 1648 Fixes schema registry request

* 1648 pom version

* 1612 add docker build

* #601 Swagger 2 PoC

* #601 Swagger 2 PoC

* #601 Swagger 2 PoC

* #1648 Updating menas-web to 3.0

* #1612 Updated npm project versions + mvn plugin

* #1612 license_check.yml

* #1612 licence check fix

Co-authored-by: Saša Zejnilović <zejnils@gmail.com>
Co-authored-by: Daniel Kavan <dk1844@gmail.com>
Co-authored-by: Jan Scherbaum <kmoj02@gmail.com>
Co-authored-by: David Benedeki <benedeki@volny.cz>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working priority: undecided Undecided priority to be assigned after discussion
Projects
None yet
Milestone
Enceladus in cloud - wave 2
Development

No branches or pull requests
2 participants
@AdrianOlosutean @Zejnilovic