Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for bigquery testing in GitHub CI via tox #203

Merged
merged 14 commits into from
Nov 22, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/ci.yml
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,9 @@ jobs:
SNOWFLAKE_DATABASE: ${{ vars.SNOWFLAKE_DATABASE }}
SNOWFLAKE_WAREHOUSE: ${{ vars.SNOWFLAKE_WAREHOUSE }}
SNOWFLAKE_SCHEMA: "integration_tests_snowflake_${{ github.run_number }}"
# bigquery
BIGQUERY_PROJECT: ${{ vars.BIGQUERY_PROJECT }}
BIGQUERY_SCHEMA: "integration_tests_bigquery_${{ github.run_number }}"
# redshift
REDSHIFT_HOST: ${{ vars.REDSHIFT_HOST }}
REDSHIFT_USER: ${{ vars.REDSHIFT_USER }}
Expand All @@ -39,3 +42,4 @@ jobs:
SNOWFLAKE_ACCOUNT: ${{ secrets.SNOWFLAKE_ACCOUNT }}
DBT_ENV_SECRET_SNOWFLAKE_PASS: ${{ secrets.SNOWFLAKE_PASS }}
DBT_ENV_SECRET_REDSHIFT_PASS: ${{ secrets.REDSHIFT_PASS }}
BIGQUERY_KEYFILE_JSON: ${{ secrets.BIGQUERY_KEYFILE_JSON }}
9 changes: 5 additions & 4 deletions integration_tests/profiles.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,11 @@ integration_tests:

bigquery:
type: bigquery
method: service-account
keyfile: "{{ env_var('BIGQUERY_SERVICE_KEY_PATH') }}"
project: "{{ env_var('BIGQUERY_TEST_DATABASE') }}"
schema: codegen_integration_tests_bigquery
method: service-account-json
keyfile_json:
"{{ env_var('BIGQUERY_KEYFILE_JSON') | as_native}}"
project: "{{ env_var('BIGQUERY_PROJECT') }}"
dataset: "{{ env_var('BIGQUERY_SCHEMA') }}"
threads: 1

snowflake:
Expand Down
2 changes: 1 addition & 1 deletion supported_adapters.env
Original file line number Diff line number Diff line change
@@ -1 +1 @@
SUPPORTED_ADAPTERS=postgres,snowflake,redshift
SUPPORTED_ADAPTERS=postgres,snowflake,redshift,bigquery
19 changes: 19 additions & 0 deletions tox.ini
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,10 @@ passenv =
SNOWFLAKE_DATABASE
SNOWFLAKE_WAREHOUSE
SNOWFLAKE_SCHEMA
# bigquery env vars
BIGQUERY_PROJECT
BIGQUERY_SCHEMA
BIGQUERY_KEYFILE_JSON
# redshift env vars
REDSHIFT_HOST
REDSHIFT_USER
Expand Down Expand Up @@ -57,6 +61,21 @@ commands =
dbt --warn-error run --target snowflake
dbt --warn-error test --target snowflake

# bigquery integration tests for centralized dbt testing
# run dbt commands directly, assumes dbt is already installed in environment
[testenv:dbt_integration_bigquery]
changedir = integration_tests
allowlist_externals =
dbt
skip_install = true
commands =
dbt --version
dbt --warn-error deps --target bigquery
dbt --warn-error run-operation create_source_table --target bigquery
dbt --warn-error seed --target bigquery --full-refresh
dbt --warn-error run --target bigquery
dbt --warn-error test --target bigquery

# redshift integration tests for centralized dbt testing
# run dbt commands directly, assumes dbt is already installed in environment
[testenv:dbt_integration_redshift]
Expand Down
Loading