Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix handling of timestamps in BigQuery insert data method #1250

Merged
merged 2 commits into from
Dec 1, 2015

Conversation

dhermes
Copy link
Contributor

@dhermes dhermes commented Nov 30, 2015

Fixes #1247.

@dhermes dhermes added the api: bigquery Issues related to the BigQuery API. label Nov 30, 2015
@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label Nov 30, 2015
@dhermes dhermes changed the title Fix handling of timestamps in Bigquery insert data method Fix handling of timestamps in BigQuery insert data method Nov 30, 2015
@dhermes
Copy link
Contributor Author

dhermes commented Nov 30, 2015

I just realized Bigtable has me trained too much. Wrote Bigquery instead of BigQuery in the commit messages.

@dhermes
Copy link
Contributor Author

dhermes commented Nov 30, 2015

FWIW I cherry-picked the second commit onto head and ran the system test with the error reported by @kamatama41 and got the following:

======================================================================
ERROR: test_load_table_then_dump_table (bigquery.TestBigQuery)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../gcloud-python/system_tests/bigquery.py", line 232, in test_load_table_then_dump_table
    rows, _, _ = table.fetch_data()
  File ".../gcloud-python/.tox/system-tests/local/lib/python2.7/site-packages/gcloud/bigquery/table.py", line 611, in fetch_data
    rows_data = _rows_from_json(response.get('rows', ()), self._schema)
  File ".../gcloud-python/.tox/system-tests/local/lib/python2.7/site-packages/gcloud/bigquery/_helpers.py", line 81, in _rows_from_json
    row_data.append(converter(cell['v'], field))
  File ".../gcloud-python/.tox/system-tests/local/lib/python2.7/site-packages/gcloud/bigquery/_helpers.py", line 42, in _datetime_from_json
    return _datetime_from_microseconds(1e6 * float(value))
  File ".../gcloud-python/.tox/system-tests/local/lib/python2.7/site-packages/gcloud/_helpers.py", line 235, in _datetime_from_microseconds
    return _EPOCH + datetime.timedelta(microseconds=value)
OverflowError: date value out of range

----------------------------------------------------------------------
Ran 11 tests in 69.796s

@tseaver
Copy link
Contributor

tseaver commented Nov 30, 2015

LGTM. OK to merge w/ pylint failing on Travis, pending #1248.

@dhermes
Copy link
Contributor Author

dhermes commented Nov 30, 2015

I'll just wait for #1248

As stated in the docs
> BigQuery stores TIMESTAMP data internally as a
> UNIX timestamp with microsecond precision
> ...Specifies the number of seconds since the epoch.
@dhermes
Copy link
Contributor Author

dhermes commented Dec 1, 2015

@tseaver Assuming your LGTM still holds after rebasing.

dhermes added a commit that referenced this pull request Dec 1, 2015
Fix handling of timestamps in BigQuery insert data method
@dhermes dhermes merged commit b4e7534 into googleapis:master Dec 1, 2015
@dhermes dhermes deleted the fix-1247 branch December 1, 2015 00:59
@kamatama41
Copy link

Great job! Thank you for fixing quickly 😄

@dhermes
Copy link
Contributor Author

dhermes commented Dec 1, 2015

Sure thing. Thanks for reporting and finding the bug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the BigQuery API. cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants