Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize list_relations_without_caching() macro to avoid calling get_relationship() on every object. #181

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

vamshikolanu
Copy link
Collaborator

@vamshikolanu vamshikolanu commented Oct 9, 2023

Describe your changes

Currently inside list_relations_without_caching() macro, we are calling get_relation_type for each and every relation. Impala doesn't have direct way to fetch all the relationships along with the table type in one query. This was causing issues where the schemas had 1000s of tables in them.

As part of this change, fetching the table type dynamically as part of the getter method. Although it isn't a standard interface provided by DBT, solving it this way for impala will reduce the number of get_relation_type() metadata calls dramatically.

Internal Jira ticket number or external issue link:

https://jira.cloudera.com/browse/DBT-744

Testing procedure/screenshots(if appropriate):

Checklist before requesting a review

  • I have performed a self-review of my code
  • I have formatted my added/modified code to follow pep-8 standards
  • I have checked suggestions from python linter to make sure code is of good quality.

@vamshikolanu vamshikolanu changed the title impala__list_relations_without_caching Optimize list_relations_without_caching() macro to avoid calling get_relationship() on every object. Oct 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant