Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: anelendata/tap-bigquery
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: master
Choose a base ref
...
head repository: meltano/tap-bigquery
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: master
Choose a head ref
Able to merge. These branches can be automatically merged.
  • 2 commits
  • 2 files changed
  • 2 contributors

Commits on Apr 11, 2023

  1. Unverified

    This user has not yet uploaded their public signing key.
    Copy the full SHA
    571e8e9 View commit details

Commits on Apr 13, 2023

  1. Merge pull request #1 from meltano/26-support-credentials-json-string

    Support providing credentials via env var.
    cjohnhanson authored Apr 13, 2023

    Unverified

    This commit is not signed, but one or more authors requires that any commit attributed to them is signed.
    Copy the full SHA
    70b1a02 View commit details
Showing with 26 additions and 4 deletions.
  1. +9 −2 README.md
  2. +17 −2 tap_bigquery/sync_bigquery.py
11 changes: 9 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -45,7 +45,7 @@ For other authentication method, please see Authentication section.

### Step 2: Install

First, make sure Python 3 is installed on your system or follow these
First, make sure Python 3 is installed on your system or follow these
installation instructions for Mac or Ubuntu.

```
@@ -62,7 +62,7 @@ pip install --no-cache-dir https://github.com/anelendata/tap-bigquery/archive/ma

### Step 1: Configure

Create a file called tap_config.json in your working directory, following
Create a file called tap_config.json in your working directory, following
config.sample.json:

```
@@ -150,6 +150,13 @@ It is recommended to use `tap-bigquery` with a service account.
- Set a `GOOGLE_APPLICATION_CREDENTIALS` environment variable on the machine,
where the value is the fully qualified path to client_secrets.json

In environments where it is preferable to supply secrets or other configuration via
environment variables or where it is not feasible to make a `client_secrets.json`
file available, you may instead provide credential as a JSON string in the
`GOOGLE_APPLICATION_CREDENTIALS_STRING` environment variable. This JSON string
should contain the same contents and be formatted the same way as the contents
of a `client_secrets.json` file.

In the testing environment, you can also manually authenticate before runnig
the tap. In this case you do not need `GOOGLE_APPLICATION_CREDENTIALS` defined:

19 changes: 17 additions & 2 deletions tap_bigquery/sync_bigquery.py
Original file line number Diff line number Diff line change
@@ -2,6 +2,7 @@
import dateutil.parser
from decimal import Decimal

from os import environ
import singer
import singer.metrics as metrics

@@ -11,6 +12,7 @@
import getschema



LOGGER = utils.get_logger(__name__)

# StitchData compatible timestamp meta data
@@ -25,6 +27,19 @@

BOOKMARK_KEY_NAME = "last_update"

SERVICE_ACCOUNT_INFO_ENV_VAR = "GOOGLE_APPLICATION_CREDENTIALS_STRING"

def get_bigquery_client():
"""Initialize a bigquery client from credentials file JSON,
if in environment, else credentials file.
Returns:
Initialized BigQuery client.
"""
credentials_json = environ.get(SERVICE_ACCOUNT_INFO_ENV_VAR)
if credentials_json:
return bigquery.Client.from_service_account_info(json.loads(credentials_json))
return bigquery.Client()

def _build_query(keys, filters=[], inclusive_start=True, limit=None):
columns = ",".join(keys["columns"])
@@ -63,7 +78,7 @@ def _build_query(keys, filters=[], inclusive_start=True, limit=None):

def do_discover(config, stream, output_schema_file=None,
add_timestamp=True):
client = bigquery.Client()
client = get_bigquery_client()

start_datetime = dateutil.parser.parse(
config.get("start_datetime")).strftime("%Y-%m-%d %H:%M:%S.%f")
@@ -144,7 +159,7 @@ def do_sync(config, state, stream):
singer.set_currently_syncing(state, stream.tap_stream_id)
singer.write_state(state)

client = bigquery.Client()
client = get_bigquery_client()
metadata = stream.metadata[0]["metadata"]
tap_stream_id = stream.tap_stream_id