Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove location and information_schema.schematas query from catalog (#2320) #2382

Merged
merged 1 commit into from
May 1, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
### Breaking changes
- Added a new dbt_project.yml version format. This emits a deprecation warning currently, but support for the existing version will be removed in a future dbt version ([#2300](https://github.com/fishtown-analytics/dbt/issues/2300), [#2312](https://github.com/fishtown-analytics/dbt/pull/2312))
- The `graph` object available in some dbt contexts now has an additional member `sources` (along side the existing `nodes`). Sources have been removed from `nodes` and added to `sources` instead ([#2312](https://github.com/fishtown-analytics/dbt/pull/2312))
- The 'location' field has been removed from bigquery catalogs ([#2382](https://github.com/fishtown-analytics/dbt/pull/2382))

### Features
- Added --fail-fast argument for dbt run and dbt test to fail on first test failure or runtime error. ([#1649](https://github.com/fishtown-analytics/dbt/issues/1649), [#2224](https://github.com/fishtown-analytics/dbt/pull/2224))
Expand Down Expand Up @@ -36,6 +37,7 @@
- Fix "dbt deps" command so it respects the "--project-dir" arg if specified. ([#2338](https://github.com/fishtown-analytics/dbt/issues/2338), [#2339](https://github.com/fishtown-analytics/dbt/issues/2339))
- On `run_cli` API calls that are passed `--vars` differing from the server's `--vars`, the RPC server rebuilds the manifest for that call. ([#2265](https://github.com/fishtown-analytics/dbt/issues/2265), [#2363](https://github.com/fishtown-analytics/dbt/pull/2363))
- Fix "Object of type Decimal is not JSON serializable" error when BigQuery queries returned numeric types in nested data structures ([#2336](https://github.com/fishtown-analytics/dbt/issues/2336), [#2348](https://github.com/fishtown-analytics/dbt/pull/2348))
- No longer query the information_schema.schemata view on bigquery ([#2320](https://github.com/fishtown-analytics/dbt/issues/2320), [#2382](https://github.com/fishtown-analytics/dbt/pull/2382))

### Under the hood
- Added more tests for source inheritance ([#2264](https://github.com/fishtown-analytics/dbt/issues/2264), [#2291](https://github.com/fishtown-analytics/dbt/pull/2291))
Expand Down
29 changes: 6 additions & 23 deletions plugins/bigquery/dbt/include/bigquery/macros/catalog.sql
Original file line number Diff line number Diff line change
Expand Up @@ -7,22 +7,7 @@
{%- else -%}

{%- set query -%}
with schemas as (

select
catalog_name as table_database,
schema_name as table_schema,
location

from {{ information_schema.replace(information_schema_view='SCHEMATA') }}
where (
{%- for schema in schemas -%}
upper(schema_name) = upper('{{ schema }}'){%- if not loop.last %} or {% endif -%}
{%- endfor -%}
)
),

tables as (
with tables as (
select
project_id as table_database,
dataset_id as table_schema,
Expand All @@ -43,7 +28,11 @@
REGEXP_EXTRACT(table_id, '^.+([0-9]{8})$') as shard_name

from {{ information_schema.replace(information_schema_view='__TABLES__') }}

where (
{%- for schema in schemas -%}
upper(dataset_id) = upper('{{ schema }}'){%- if not loop.last %} or {% endif -%}
{%- endfor -%}
)
),

extracted as (
Expand Down Expand Up @@ -171,11 +160,6 @@
coalesce(columns.column_type, '<unknown>') as column_type,
columns.column_comment,

'Location' as `stats__location__label`,
location as `stats__location__value`,
'The geographic location of this table' as `stats__location__description`,
location is not null as `stats__location__include`,

'Shard count' as `stats__date_shards__label`,
table_shards.shard_count as `stats__date_shards__value`,
'The number of date shards in this table' as `stats__date_shards__description`,
Expand Down Expand Up @@ -215,7 +199,6 @@
-- sure that column metadata is picked up through the join. This will only
-- return the column information for the "max" table in a date-sharded table set
from unsharded_tables
left join schemas using(table_database, table_schema)
left join columns using (relation_id)
left join column_stats using (relation_id)
{%- endset -%}
Expand Down
27 changes: 11 additions & 16 deletions test/integration/029_docs_generate_tests/test_docs_generate.py
Original file line number Diff line number Diff line change
Expand Up @@ -253,22 +253,8 @@ def _snowflake_stats(self):
}

def _bigquery_stats(self, is_table, partition=None, cluster=None):
stats = {
'has_stats': {
'id': 'has_stats',
'label': 'Has Stats?',
'value': True,
'description': 'Indicates whether there are statistics for this table',
'include': False,
},
'location': {
'id': 'location',
'label': 'Location',
'value': 'US',
'description': 'The geographic location of this table',
'include': True,
}
}
stats = {}

if is_table:
stats.update({
'num_bytes': {
Expand Down Expand Up @@ -309,6 +295,15 @@ def _bigquery_stats(self, is_table, partition=None, cluster=None):
}
})

has_stats = {
'id': 'has_stats',
'label': 'Has Stats?',
'value': bool(stats),
'description': 'Indicates whether there are statistics for this table',
'include': False,
}
stats['has_stats'] = has_stats

return stats

def _expected_catalog(self, id_type, text_type, time_type, view_type,
Expand Down