Feature/1733 lineage dumper #1742
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#1110: Make Standardization and Conformance use Spline 0.4.x * Change Spline version * Make **Standardization** and **Confluence** depend and init the new Spline module instead of the old ones * Replacing Spline properties in templates * Using a different JSON library (instead of the one transitionally ingested from old Spline) * Application properties templates adjutsted * Helper scripts updated * README.md updated (addressing also changes in Menas and Oozie from other PR) * Missing copyright headers Co-authored-by: Georgi Chochov <g.chochov@gmail.com>
* Server-side api endpoints to provide new configuration * Server-side controller to serve the Spline WebJar * Client-side changes to gather the lineage id and display the lineage new UI * Added dependency to include the webjar * Header to allow iframe display `<frame-options policy="SAMEORIGIN"/>` * fixed `style.css`
* Rebasing to current `Develop`
* Unused import removed
…ncies * Version names sorted alphabetically
# Conflicts (resolved): # pom.xml # scripts/bash/run_enceladus.sh # spark-jobs/pom.xml
* Create license_check.yml
% Conflicts: % menas/src/main/scala/za/co/absa/enceladus/menas/MvcConfig.scala % menas/src/main/scala/za/co/absa/enceladus/menas/services/RunService.scala % menas/src/test/resources/application.properties % menas/ui/css/style.css % pom.xml % spark-jobs/pom.xml % spark-jobs/src/main/scala/za/co/absa/enceladus/conformance/DynamicConformanceJob.scala % spark-jobs/src/main/scala/za/co/absa/enceladus/standardization/StandardizationJob.scala % spark-jobs/src/test/scala/za/co/absa/enceladus/conformance/interpreter/InterpreterSuite.scala % utils/src/test/scala/za/co/absa/enceladus/utils/validation/field/ScalarFieldValidatorSuite.scala
…ineage-dumper % Conflicts: % pom.xml
…za/co/absa/commons/config/ConfigurationImplicits$ConfigurationRequiredWrapper$)
* Add e2e tests into examples * Fix style * Examples - Fix comments * Apply suggestions from code review Co-authored-by: veprokendlo <68275331+veprokendlo@users.noreply.github.com> * Update vars in test jasons and other small fixes * new test for MappingCR and other small fixes * repair key address * typo remove * add recordId conf into test jasons and update readme Co-authored-by: veprokendlo <68275331+veprokendlo@users.noreply.github.com> Co-authored-by: Veronika Huvarova <vhuvarova@gmail.com> Co-authored-by: Veronika <56251097+HuvarVer@users.noreply.github.com>
dk1844
requested review from
AdrianOlosutean,
benedeki,
DzMakatun,
HuvarVer and
lokm01
as code owners
|
Kudos, SonarCloud Quality Gate passed!
|
|
Closed in favor of #1766 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
(blocked by SNAPSHOT version of spline agent
and SNAPSHOT version of Absa commons) for the time beingUsing a SNAPSHOT spline agent from AbsaOSS/spline-spark-agent#197, Enceladus's
spark-jobsrun is able to produce a_LINEAGEfile on S3.hdfsdispatcher must be enabled. (Using `spark-submit ... --conf "spark.driver.extraJavaOptions=-Dspark.spline.lineageDispatcher=hdfs ... "```)httpdispatcher (responsible for sending data to spline server/ui), we may want to use the composite dispatcher to get both:... -Dspark.spline.lineageDispatcher=composite -Dspline.lineageDispatcher.composite.dispatchers=http,hdfsTest run
This was actually built into an Enceladus
spark-jobsfat jar and ran on EMR for Standardization:(values of
MENAS_URLandSPLINE_URL; andmy-example-bucket123are not actual values used)Closes #1733.