Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot generate external model for BigQuery information schema #3317

Closed
ncbkr opened this issue Oct 31, 2024 · 0 comments · Fixed by #3332
Closed

Cannot generate external model for BigQuery information schema #3317

ncbkr opened this issue Oct 31, 2024 · 0 comments · Fixed by #3332
Assignees
Labels
Bug Something isn't working Engine: BigQuery issues related to BigQuery

Comments

@ncbkr
Copy link

ncbkr commented Oct 31, 2024

We have a model that is querying the BigQuery information schema. In contrast to regular tables living in datasets within projects, the INFORMATION_SCHEMA follows a different syntax that does not have to comply with the usual <project>.<dataset>.<table> format. E.g. this would be valid identifier for an information schema table:

myProject.`region-us`.INFORMATION_SCHEMA.TABLES` 

We have a query like the following:

SELECT *
FROM `region-eu`.INFORMATION_SCHEMA.JOBS

When running sqlmesh create_external_models, we receive the following warning

2024-10-31 15:08:10,605 - ThreadPoolExecutor-0_0 - sqlmesh.core.schema_loader - WARNING - Unable to get schema for '"region-eu"."INFORMATION_SCHEMA"."JOBS"': '404 GET https://bigquery.googleapis.com/bigquery/v2/projects/region-eu/datasets/INFORMATION_SCHEMA/tables/JOBS?prettyPrint=false: Project region-eu is not found. Make sure it references valid GCP project that hasn't been deleted.; Project id: region-eu'. (schema_loader.py:62)

The warning indicates that SQLMesh is interpreting region-eu as the project name, which is not true for information schema tables.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something isn't working Engine: BigQuery issues related to BigQuery
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants