Feautre/1733 lineage dumper - 2nd edition

[dk1844]

This PR should effectively replace #1742 - new cherry-picked on develop-ver3.0 basis without any spline-0.4-6 integration baggage.

Using a 0.6.0-SNAPSHOT spline agent from AbsaOSS/spline-spark-agent.

Test run

• needs compatible develop-ver3.0 means on AWS to run

export MENAS_URL="https://some.menas.on.aws"
export SPLINE_URL="https://some.spline.0.6+.backend/producer"

spark-submit --class za.co.absa.enceladus.standardization.StandardizationJob \
--conf "spark.driver.extraJavaOptions=-Dmenas.rest.uri=$MENAS_URL -Dstandardized.hdfs.path=s3://my-example-bucket123/superhero2/std/standardized-{0}-{1}-{2}-{3} -Dspline.producer.url=$SPLINE_URL -Dspline.lineageDispatcher=composite -Dspline.lineageDispatcher.composite.dispatchers=http,hdfs " \
~/enceladus/spark-jobs-3.0.0-SNAPSHOT.jar \
--menas-credentials-file ~/enceladus/svc-aws-credentials.properties \
--dataset-name SuperheroesS3 --dataset-version 1 --report-date 2020-08-06 --report-version 1 \
--raw-format csv --header true 

$ aws --profile saml s3 ls --recursive s3://my-example-bucket123/superhero2/std
2021-04-23 14:35:48       2225 superhero2/std/standardized-SuperheroesS3-1-2020-08-06-1/_INFO
2021-04-23 14:35:43     127725 superhero2/std/standardized-SuperheroesS3-1-2020-08-06-1/_LINEAGE
2021-04-23 14:35:35          0 superhero2/std/standardized-SuperheroesS3-1-2020-08-06-1/_SUCCESS
2021-04-23 14:35:35      45229 superhero2/std/standardized-SuperheroesS3-1-2020-08-06-1/part-00000-c16c8a6d-84b3-46bd-8a
51-f86f2ea14071-c000.snappy.parquet

(values of MENAS_URL and SPLINE_URL; and my-example-bucket123 are not actual values used)

• testran again @ 2021-05-25, reconfirmed to work - _LINEAGE file is present on S3.

Closes #1733.

RN suggestion

• Spline agent 0.6.0 integrated - support for _LINEAGE file generation is present

[AdrianOlosutean]

Point/base this PR to develop-ver3.0?

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[AdrianOlosutean]

Looks good(didn't get to test though)