Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

archives: do not create existing schema (#758) #1398

Merged
merged 2 commits into from
Apr 26, 2019
Merged

archives: do not create existing schema (#758) #1398

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2efae5a
give bigquery list_schemas/check_schema_exist macros
Apr 12, 2019
cc4f285
if the schema exists, do not try to create it
Apr 12, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion core/dbt/include/global_project/macros/materializations/archive/archive.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -253,7 +253,14 @@
{%- set target_table = model.get('alias', model.get('name')) -%}
{%- set strategy = config.get('strategy') -%}

{{ create_schema(target_database, target_schema) }}
{% set information_schema = api.Relation.create(
database=target_database,
schema=target_schema,
identifier=target_table).information_schema() %}

{% if not check_schema_exists(information_schema, target_schema) %}
{{ create_schema(target_database, target_schema) }}
{% endif %}

{%- set target_relation = adapter.get_relation(
database=target_database,
Expand Down
6 changes: 6 additions & 0 deletions plugins/bigquery/dbt/adapters/bigquery/impl.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ def rename_relation(self, from_relation, to_relation):
'`rename_relation` is not implemented for this adapter!'
)

@available
def list_schemas(self, database):
conn = self.connections.get_thread_connection()
client = conn.handle
Expand All @@ -84,6 +85,11 @@ def list_schemas(self, database):
include_all=True)
return [ds.dataset_id for ds in all_datasets]

@available
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this mean that we'll need to do something similar for the Presto and Spark plugins?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This @available exists to make the relation available to bq's macros so it can implement this macro, because on bigquery that functionality is implemented via the connection.

Spark is a SQLAdapter and so it already has the macro available. Though it does look like it's incomplete?

We currently don't support archives on presto, so we don't have to worry about that at all.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hah, dang. Ok, thanks for the info, we can handle Spark separately from this for sure

def check_schema_exists(self, database, schema):
superself = super(BigQueryAdapter, self)
return superself.check_schema_exists(database, schema)

def get_columns_in_relation(self, relation):
try:
table = self.connections.get_bq_table(
Expand Down
10 changes: 10 additions & 0 deletions plugins/bigquery/dbt/include/bigquery/macros/adapters.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,3 +67,13 @@
{% macro bigquery__current_timestamp() -%}
CURRENT_TIMESTAMP()
{%- endmacro %}


{% macro bigquery__list_schemas(database) %}
{{ return(adapter.list_schemas()) }}
{% endmacro %}


{% macro bigquery__check_schema_exists(information_schema, schema) %}
{{ return(adapter.check_schema_exists(information_schema.database, schema)) }}
{% endmacro %}