Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ingest): add underlying platform for glue #3035

Merged

Conversation

aseembansal-gogo
Copy link
Contributor

@aseembansal-gogo aseembansal-gogo commented Aug 5, 2021

This is in place of earlier PR #2985.

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable)

@aseembansal-gogo
Copy link
Contributor Author

I think the other changes might be because in this #3030 the commands were not run.

Are the tests on github different compared to what is mentioned in https://datahubproject.io/docs/metadata-ingestion/developing#testing @shirshanka ?

@shirshanka
Copy link
Contributor

Hi @aseembansal-gogo

Platforms are special entities, they indicate technologies which have logos, and can have UI behavior attached to them. I don't think we want to allow unfettered ability to replace platform names with other strings.

In fact we have allowed this in some other sources (like looker) and want to remove this capability.

We will introduce the concept of "instances" which will allow you to express free-form labels like "data-warehouse" to attach to these systems.

I would like to propose a modification to your current approach where:

  1. In the special case where glue is attached to an athena-only warehouse, one can replace the platform glue with the underlying_platform athena (hence staying away from the verbiage platform_alias)
  2. This list of replacements allowed should be a controlled list of platforms, which will start with just 1 option for now: athena

Hope this will unblock you.

@aseembansal-gogo aseembansal-gogo changed the title feat(ingest): add platform alias for glue and database alias for mysql feat(ingest): add underlying platform for glue and database alias for mysql Aug 6, 2021
Copy link
Contributor

@shirshanka shirshanka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for making the changes.

@aseembansal-gogo aseembansal-gogo changed the title feat(ingest): add underlying platform for glue and database alias for mysql feat(ingest): add underlying platform for glue Aug 6, 2021
Copy link
Contributor

@shirshanka shirshanka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@shirshanka shirshanka merged commit ac345e0 into datahub-project:master Aug 6, 2021
@aseembansal-gogo aseembansal-gogo deleted the alias_glue_mysql branch August 6, 2021 17:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants