Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spline _LINEAGE dumper to work on S3 #1733

Closed
benedeki opened this issue Mar 16, 2021 · 1 comment · Fixed by #1766
Closed

Spline _LINEAGE dumper to work on S3 #1733

benedeki opened this issue Mar 16, 2021 · 1 comment · Fixed by #1766
Assignees
@dk1844
Labels
3rd party issue Issue not our own in origin but we are slowed down by them Conformance Conformance Job affected feature New feature priority: high Critical to the health of the project Standardization Standardization Job affected

Comments

@benedeki
Copy link
Collaborator

benedeki commented Mar 16, 2021 •
edited
Loading

Background

Spline is able to write the lineage to HDFS. With migration to AWS we need to ensure this feature works on S3 as well.

Feature

Enhance, if needed, Spline Dumper and Enceladus to write to S3.

Proposed Solution

Perhaps could be solved similarly as Atum, with optional libraries to include when S3 is to be supported.
@benedeki benedeki added 3rd party issue Issue not our own in origin but we are slowed down by them Conformance Conformance Job affected feature New feature priority: high Critical to the health of the project Standardization Standardization Job affected labels Mar 17, 2021
dk1844 added a commit that referenced this issue Mar 23, 2021
@dk1844
#1733 snaphotted version of spline-agent with s3 dumper implementatio…
aba7bfe 
…n - todo: test
dk1844 added a commit that referenced this issue Mar 23, 2021
@dk1844
#1733 build fix after merging develop
1584076
dk1844 added a commit that referenced this issue Mar 26, 2021
@dk1844
#1733 absa commons update (otherwise java.lang.NoClassDefFoundError: …
eed600b 
…za/co/absa/commons/config/ConfigurationImplicits$ConfigurationRequiredWrapper$)
dk1844 added a commit that referenced this issue Mar 26, 2021
@dk1844
#1733 json4s provided
2862018
@dk1844 dk1844 mentioned this issue Mar 29, 2021
Closed
dk1844 added a commit that referenced this issue Apr 7, 2021
@dk1844
#1733 s3Location moved to absa commons
6d0dd62
dk1844 added a commit that referenced this issue Apr 12, 2021
@dk1844
#1733/1732 buildfix
da035b5
dk1844 added a commit that referenced this issue Apr 12, 2021
@dk1844
#1733/1732 cleanup
aa6bb58
dk1844 added a commit that referenced this issue Apr 23, 2021
@dk1844
#1733 cherrypicked version: commons 0.0.27, atum 3.5.0, spline agent …
1ef5c94 
…0.6.0-snapshot
@dk1844 dk1844 mentioned this issue Apr 23, 2021
Merged
dk1844 added a commit that referenced this issue Apr 26, 2021
@dk1844
#1733 PR touchup: maven.shade.version version revert to 3.2.1
efb3edd
@benedeki benedeki mentioned this issue Apr 26, 2021
Closed
dk1844 added a commit that referenced this issue May 24, 2021
@dk1844
#1733 final version 0.6.0 of spline spark agent used
e0869ae
dk1844 added a commit that referenced this issue May 25, 2021
@dk1844
Feautre/1733 lineage dumper - 2nd edition (#1766)
70999f5 
* #1733 final version 0.6.0 of spline spark agent used
* #1733 cherrypicked version: commons 0.0.27, atum 3.5.0, spline agent 0.6.0-snapshot
* #1733 PR touchup: maven.shade.version version revert to 3.2.1
@dk1844
Copy link
Contributor

dk1844 commented May 25, 2021

Release notes
Spline agent 0.6.0 integrated - support for _LINEAGE file generation is present.

@dk1844 dk1844 closed this as completed May 25, 2021
@dk1844 dk1844 linked a pull request May 25, 2021 that will close this issue
Feautre/1733 lineage dumper - 2nd edition #1766
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dk1844 dk1844

Labels
3rd party issue Issue not our own in origin but we are slowed down by them Conformance Conformance Job affected feature New feature priority: high Critical to the health of the project Standardization Standardization Job affected
Projects
None yet
Milestone
No milestone
2 participants
@dk1844 @benedeki