Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BigQuery: '_time_from_json' raises for values w/ sub-second precision #5750

Closed
cpcloud opened this issue Aug 6, 2018 · 5 comments
Closed
Assignees
Labels
api: bigquery Issues related to the BigQuery API. api: core priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@cpcloud
Copy link

cpcloud commented Aug 6, 2018

  • OS: Arch Linux
  • relevant conda versions:
# packages in environment at /home/phillip/miniconda3/envs/ibis36:
#
# Name                    Version                   Build  Channel
google-cloud-bigquery     1.4.0                      py_0    conda-forge
google-cloud-core         0.28.1                     py_0    conda-forge
  • Steps to reproduce:
import google.cloud.bigquery as bq
client = bq.Client('ibis-gbq')
sql = """
SELECT TIME(`creation_date`) AS `created_time`
FROM `bigquery-public-data.stackoverflow.posts_questions`
"""
query = client.query(sql)
df = query.result().to_dataframe()
  • Traceback:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-27-caeea53c88e2> in <module>()
----> 1 df = query.result().to_dataframe()

~/miniconda3/envs/ibis36/lib/python3.6/site-packages/google/cloud/bigquery/table.py in to_dataframe(self)
   1145
   1146         column_headers = [field.name for field in self.schema]
-> 1147         rows = [row.values() for row in iter(self)]
   1148
   1149         return pandas.DataFrame(rows, columns=column_headers)

~/miniconda3/envs/ibis36/lib/python3.6/site-packages/google/cloud/bigquery/table.py in <listcomp>(.0)
   1145
   1146         column_headers = [field.name for field in self.schema]
-> 1147         rows = [row.values() for row in iter(self)]
   1148
   1149         return pandas.DataFrame(rows, columns=column_headers)

~/miniconda3/envs/ibis36/lib/python3.6/site-packages/google/api_core/page_iterator.py in _items_iter(self)
    198         """Iterator for each item returned."""
    199         for page in self._page_iter(increment=False):
--> 200             for item in page:
    201                 self.num_results += 1
    202                 yield item

~/miniconda3/envs/ibis36/lib/python3.6/site-packages/google/api_core/page_iterator.py in next(self)
    123         """Get the next value in the page."""
    124         item = six.next(self._item_iter)
--> 125         result = self._item_to_value(self._parent, item)
    126         # Since we've successfully got the next value from the
    127         # iterator, we update the number of remaining.

~/miniconda3/envs/ibis36/lib/python3.6/site-packages/google/cloud/bigquery/table.py in _item_to_row(iterator, resource)
   1314     :returns: The next row in the page.
   1315     """
-> 1316     return Row(_helpers._row_tuple_from_json(resource, iterator.schema),
   1317                iterator._field_to_index)
   1318

~/miniconda3/envs/ibis36/lib/python3.6/site-packages/google/cloud/bigquery/_helpers.py in _row_tuple_from_json(row, schema)
    210                              for item in cell['v']])
    211         else:
--> 212             row_data.append(converter(cell['v'], field))
    213
    214     return tuple(row_data)

~/miniconda3/envs/ibis36/lib/python3.6/site-packages/google/cloud/bigquery/_helpers.py in _time_from_json(value, field)
    144     if _not_null(value, field):
    145         # value will be a string, in HH:MM:SS form.
--> 146         return _time_from_iso8601_time_naive(value)
    147
    148

~/miniconda3/envs/ibis36/lib/python3.6/site-packages/google/cloud/_helpers.py in _time_from_iso8601_time_naive(value)
    259
    260     """
--> 261     return datetime.datetime.strptime(value, '%H:%M:%S').time()
    262
    263

~/miniconda3/envs/ibis36/lib/python3.6/_strptime.py in _strptime_datetime(cls, data_string, format)
    563     """Return a class cls instance based on the input string and the
    564     format string."""
--> 565     tt, fraction = _strptime(data_string, format)
    566     tzname, gmtoff = tt[-2:]
    567     args = tt[:6] + (fraction,)

~/miniconda3/envs/ibis36/lib/python3.6/_strptime.py in _strptime(data_string, format)
    363     if len(data_string) != found.end():
    364         raise ValueError("unconverted data remains: %s" %
--> 365                           data_string[found.end():])
    366
    367     iso_year = year = None

ValueError: unconverted data remains: .590000
@tseaver tseaver added the api: bigquery Issues related to the BigQuery API. label Aug 6, 2018
@tseaver
Copy link
Contributor

tseaver commented Aug 6, 2018

I can reproduce locally, with the correction that the field name in the query should be creation_date, rather than created_date.

@tseaver tseaver added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. priority: p2 Moderately-important priority. Fix may not be included in next release. labels Aug 6, 2018
@cpcloud
Copy link
Author

cpcloud commented Aug 6, 2018

@tseaver Whoops, thanks. Fixed.

@tseaver tseaver changed the title BigQuery: time type conversion doesn't work for values that have more precision than seconds BigQuery: '_time_from_json' raises for values w/ sub-second precision Aug 6, 2018
@tseaver
Copy link
Contributor

tseaver commented Aug 6, 2018

@cpcloud Thanks very much for the report. #5753 will address the issue for BigQuery. I will leave this issue open to propagate the fix to 'google.cloud._helpers._time_from_iso8601_time_naive'.

tseaver added a commit that referenced this issue Aug 6, 2018
See #5750.

The issue remains in 'google.cloud._helpers._time_from_iso8601_time_naive',
but I don't want to have BigQuery wait on a release of core.
@cpcloud
Copy link
Author

cpcloud commented Aug 6, 2018

@tseaver Excellent! Thanks for addressing the issue.

@tswast
Copy link
Contributor

tswast commented Aug 10, 2018

Thanks @cpcloud and @tseaver for the report and the fix!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the BigQuery API. api: core priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Development

No branches or pull requests

3 participants