Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enceladus #1894: S3 support for HdfsDataLineageWriter #953

Closed
wants to merge 1 commit into from
Closed

Enceladus #1894: S3 support for HdfsDataLineageWriter #953

wants to merge 1 commit into from

Conversation

dk1844
Copy link
Contributor

@dk1844 dk1844 commented Sep 6, 2021 •
edited
Loading

Backported AbsaOSS/spline-spark-agent#195 for version 0.3 requested in AbsaOSS/enceladus#1894

Test-run

  • built this snapshot version of spline, built custom Enceladus 2.24.0-SNAPSHOT from master with this custom dependency and ran on EMR with dataset's conformed publish path being located on S3. The result now includes a _LINEAGE file:
$ aws --profile saml s3 ls s3://<my bucket name here>/publish/superhero/ --recursive
2021-09-08 12:44:03       4048 publish/superhero/enceladus_info_date=2020-08-06/enceladus_info_version=1/_INFO
2021-09-08 12:44:01      10600 publish/superhero/enceladus_info_date=2020-08-06/enceladus_info_version=1/_LINEAGE
2021-09-08 12:44:00          0 publish/superhero/enceladus_info_date=2020-08-06/enceladus_info_version=1/_SUCCESS
2021-09-08 12:44:00      46554 publish/superhero/enceladus_info_date=2020-08-06/enceladus_info_version=1/part-00000-e81b
a7e6-ee18-4fd7-8bfa-76759eb00907-c000.snappy.parquet

@sonarcloud
Copy link

sonarcloud bot commented Sep 6, 2021

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@dk1844 dk1844 marked this pull request as ready for review September 6, 2021 14:32
@dk1844 dk1844 marked this pull request as draft September 6, 2021 14:32
@dk1844 dk1844 marked this pull request as ready for review September 9, 2021 10:36
Zejnilovic
Zejnilovic approved these changes Sep 9, 2021
View reviewed changes
Copy link
Contributor

@Zejnilovic Zejnilovic left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like it. I leveraged the "same" code 2 days ago in Hermes myself so I know it works

AdrianOlosutean
AdrianOlosutean approved these changes Sep 10, 2021
View reviewed changes
Copy link
Contributor

@AdrianOlosutean AdrianOlosutean left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

@dk1844 dk1844 mentioned this pull request Sep 10, 2021
Merged
@dk1844
Copy link
Contributor Author

dk1844 commented Sep 10, 2021

Closing in favor of AbsaOSS/enceladus#1912.

@dk1844 dk1844 closed this Sep 10, 2021
@wajda wajda deleted the feature/enceladus1894-s3supportForV0.3 branch October 7, 2021 21:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Zejnilovic Zejnilovic Zejnilovic approved these changes

@AdrianOlosutean AdrianOlosutean AdrianOlosutean approved these changes

@benedeki benedeki Awaiting requested review from benedeki

@wajda wajda Awaiting requested review from wajda

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@dk1844 @AdrianOlosutean @Zejnilovic