Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bigquery optimise queries for JOBS_BY_PROJECT table. #14099

Closed
ulixius9 opened this issue Nov 24, 2023 · 0 comments · Fixed by #14106
Closed

Bigquery optimise queries for JOBS_BY_PROJECT table. #14099

ulixius9 opened this issue Nov 24, 2023 · 0 comments · Fixed by #14106
Assignees
@ulixius9
Labels
bug Something isn't working Ingestion

Comments

@ulixius9
Copy link
Member

Affected module
Ingestion Framework

Describe the bug
When querying the table JOBS_BY_PROJECT to fetch the queries for analyzing usage or lineage we are scanning the entire table which can be costly.

The table is partitioned by creation_time and clustered by project id, when querying this table we should include these filters ref.

queries that need to be updated.
@ulixius9 ulixius9 added bug Something isn't working Ingestion labels Nov 24, 2023
@ulixius9 ulixius9 self-assigned this Nov 24, 2023
ulixius9 added a commit that referenced this issue Nov 24, 2023
@ulixius9
Fix #14099: fix bigquery test connection & sp query
d723cc8
@ulixius9 ulixius9 mentioned this issue Nov 24, 2023
Merged
9 tasks
pmbrull pushed a commit that referenced this issue Nov 28, 2023
@ulixius9
#14099: fix bigquery test connection & SP query (#14106)
49a5557 
* Fix #14099: fix bigquery test connection & sp query

* fix unquote condition

* pyformat
@harshach harshach moved this to Done in Release 1.2.3 Nov 30, 2023
OnkarVO7 pushed a commit that referenced this issue Dec 5, 2023
@ulixius9 @OnkarVO7
#14099: fix bigquery test connection & SP query (#14106)
efb422d 
* Fix #14099: fix bigquery test connection & sp query

* fix unquote condition

* pyformat
MrVinegar pushed a commit to MrVinegar/OpenMetadata that referenced this issue Dec 15, 2023
@ulixius9
open-metadata#14099: fix bigquery test connection & SP query (open-me…
bba11cd 
…tadata#14106)

* Fix open-metadata#14099: fix bigquery test connection & sp query

* fix unquote condition

* pyformat
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ulixius9 ulixius9

Labels
bug Something isn't working Ingestion
Projects
No open projects
Release 1.2.3
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

#14099: fix bigquery test connection & SP query open-metadata/OpenMetadata
1 participant
@ulixius9