Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BigQuery check if schema already exists before trying to create it #220

Merged
merged 2 commits into from
Aug 30, 2023
Merged

BigQuery check if schema already exists before trying to create it #220

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ca8771d
fix check schema
Jun 21, 2023
e523ef6
fix when we do not need to update the schema
Jun 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 16 additions & 4 deletions macros/plugins/bigquery/create_external_schema.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,21 @@
{%- endif -%}
{%- endset -%}

{%- set ddl -%}
create schema if not exists {{ fqn }}
{%- endset -%}
{% set schema_exists_query %}
select * from {{ source_node.database }}.INFORMATION_SCHEMA.SCHEMATA where schema_name = '{{ source_node.schema }}' limit 1
{% endset %}
{% if execute %}
{% set schema_exists = run_query(schema_exists_query)|length > 0 %}
{% else %}
{% set schema_exists = false %}
{% endif %}

{{ return(ddl) }}
{%- if not schema_exists -%}
{%- set ddl -%}
create schema if not exists {{ fqn }}
{%- endset -%}
{{ return(ddl) }}
{%- else -%}
{{ return('') }}
{% endif %}
{%- endmacro -%}
14 changes: 10 additions & 4 deletions macros/plugins/bigquery/get_external_build_plan.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,16 @@
{% set create_or_replace = (old_relation is none or var('ext_full_refresh', false)) %}

{% if create_or_replace %}
{% set build_plan = build_plan + [
dbt_external_tables.create_external_schema(source_node),
dbt_external_tables.create_external_table(source_node)
] %}
{% if not dbt_external_tables.create_external_schema(source_node)|length %}
{% set build_plan = build_plan + [
dbt_external_tables.create_external_table(source_node)
] %}
{% else %}
{% set build_plan = build_plan + [
dbt_external_tables.create_external_schema(source_node),
dbt_external_tables.create_external_table(source_node)
] %}
{% endif %}
{% else %}
{% set build_plan = build_plan + dbt_external_tables.refresh_external_table(source_node) %}
{% endif %}
Expand Down
2 changes: 1 addition & 1 deletion run_test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/bin/bash

echo "Setting up virtual environment"
echo "Setting up virtual environment for dbt-$1"
VENV="venv/bin/activate"

if [[ ! -f $VENV ]]; then
Expand Down