Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: anelendata/tap-bigquery
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: master
Choose a base ref
...
head repository: meltano/tap-bigquery
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: master
Choose a head ref
Able to merge. These branches can be automatically merged.
  • 2 commits
  • 2 files changed
  • 2 contributors

Commits on Apr 11, 2023

  1. Copy the full SHA
    571e8e9 View commit details

Commits on Apr 13, 2023

  1. Merge pull request #1 from meltano/26-support-credentials-json-string

    Support providing credentials via env var.
    cjohnhanson authored Apr 13, 2023
    Copy the full SHA
    70b1a02 View commit details
Showing with 26 additions and 4 deletions.
  1. +9 −2 README.md
  2. +17 −2 tap_bigquery/sync_bigquery.py
11 changes: 9 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -45,7 +45,7 @@ For other authentication method, please see Authentication section.

### Step 2: Install

First, make sure Python 3 is installed on your system or follow these
First, make sure Python 3 is installed on your system or follow these
installation instructions for Mac or Ubuntu.

```
@@ -62,7 +62,7 @@ pip install --no-cache-dir https://github.com/anelendata/tap-bigquery/archive/ma

### Step 1: Configure

Create a file called tap_config.json in your working directory, following
Create a file called tap_config.json in your working directory, following
config.sample.json:

```
@@ -150,6 +150,13 @@ It is recommended to use `tap-bigquery` with a service account.
- Set a `GOOGLE_APPLICATION_CREDENTIALS` environment variable on the machine,
where the value is the fully qualified path to client_secrets.json

In environments where it is preferable to supply secrets or other configuration via
environment variables or where it is not feasible to make a `client_secrets.json`
file available, you may instead provide credential as a JSON string in the
`GOOGLE_APPLICATION_CREDENTIALS_STRING` environment variable. This JSON string
should contain the same contents and be formatted the same way as the contents
of a `client_secrets.json` file.

In the testing environment, you can also manually authenticate before runnig
the tap. In this case you do not need `GOOGLE_APPLICATION_CREDENTIALS` defined:

19 changes: 17 additions & 2 deletions tap_bigquery/sync_bigquery.py
Original file line number Diff line number Diff line change
@@ -2,6 +2,7 @@
import dateutil.parser
from decimal import Decimal

from os import environ
import singer
import singer.metrics as metrics

@@ -11,6 +12,7 @@
import getschema



LOGGER = utils.get_logger(__name__)

# StitchData compatible timestamp meta data
@@ -25,6 +27,19 @@

BOOKMARK_KEY_NAME = "last_update"

SERVICE_ACCOUNT_INFO_ENV_VAR = "GOOGLE_APPLICATION_CREDENTIALS_STRING"

def get_bigquery_client():
"""Initialize a bigquery client from credentials file JSON,
if in environment, else credentials file.
Returns:
Initialized BigQuery client.
"""
credentials_json = environ.get(SERVICE_ACCOUNT_INFO_ENV_VAR)
if credentials_json:
return bigquery.Client.from_service_account_info(json.loads(credentials_json))
return bigquery.Client()

def _build_query(keys, filters=[], inclusive_start=True, limit=None):
columns = ",".join(keys["columns"])
@@ -63,7 +78,7 @@ def _build_query(keys, filters=[], inclusive_start=True, limit=None):

def do_discover(config, stream, output_schema_file=None,
add_timestamp=True):
client = bigquery.Client()
client = get_bigquery_client()

start_datetime = dateutil.parser.parse(
config.get("start_datetime")).strftime("%Y-%m-%d %H:%M:%S.%f")
@@ -144,7 +159,7 @@ def do_sync(config, state, stream):
singer.set_currently_syncing(state, stream.tap_stream_id)
singer.write_state(state)

client = bigquery.Client()
client = get_bigquery_client()
metadata = stream.metadata[0]["metadata"]
tap_stream_id = stream.tap_stream_id