Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DB-API depends on pyarrow when decimal query parameters are used #549

Closed
tswast opened this issue Mar 12, 2021 · 0 comments · Fixed by #551
Closed

DB-API depends on pyarrow when decimal query parameters are used #549

tswast opened this issue Mar 12, 2021 · 0 comments · Fixed by #551
Assignees
Labels
priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@tswast
Copy link
Contributor

tswast commented Mar 12, 2021

It seems we introduced a hard dependency on pyarrow to detect data types of query parameters in the DB-API module. While pyarrow is recommended as it's needed to use the BigQuery Storage API, I don't think we want to depend on it for basic functionality.

I believe we added this hard dependency here: https://github.com/googleapis/python-bigquery/pull/527/files#diff-2221cd9b6b85f2ed1f943c474e8a8628e6ca28ec9a96c0aa5873cd6a66826bdeR193

We use pyarrow to disambiguate between NUMERIC and BIGNUMERIC. I'm not seeing a good way to check the precision / scale in the decimal type, so in the case that pyarrow is not available, we should guess the type is NUMERIC.

Stack trace

__________________________________________________________________________________ test_dml ___________________________________________________________________________________
test/test_sqlalchemy_bigquery.py:430: in test_dml
    engine.execute(table_dml.insert(ONE_ROW_CONTENTS_DML))
../../miniconda3/envs/pybigquery/lib/python3.9/site-packages/sqlalchemy/engine/base.py:2235: in execute
    return connection.execute(statement, *multiparams, **params)
../../miniconda3/envs/pybigquery/lib/python3.9/site-packages/sqlalchemy/engine/base.py:1011: in execute
    return meth(self, multiparams, params)
../../miniconda3/envs/pybigquery/lib/python3.9/site-packages/sqlalchemy/sql/elements.py:298: in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
../../miniconda3/envs/pybigquery/lib/python3.9/site-packages/sqlalchemy/engine/base.py:1124: in _execute_clauseelement
    ret = self._execute_context(
../../miniconda3/envs/pybigquery/lib/python3.9/site-packages/sqlalchemy/engine/base.py:1316: in _execute_context
    self._handle_dbapi_exception(
../../miniconda3/envs/pybigquery/lib/python3.9/site-packages/sqlalchemy/engine/base.py:1514: in _handle_dbapi_exception
    util.raise_(exc_info[1], with_traceback=exc_info[2])
../../miniconda3/envs/pybigquery/lib/python3.9/site-packages/sqlalchemy/util/compat.py:182: in raise_
    raise exception
../../miniconda3/envs/pybigquery/lib/python3.9/site-packages/sqlalchemy/engine/base.py:1276: in _execute_context
    self.dialect.do_execute(
../../miniconda3/envs/pybigquery/lib/python3.9/site-packages/sqlalchemy/engine/default.py:608: in do_execute
    cursor.execute(statement, parameters)
../../miniconda3/envs/pybigquery/lib/python3.9/site-packages/google/cloud/bigquery/dbapi/_helpers.py:263: in with_closed_check
    return method(self, *args, **kwargs)
../../miniconda3/envs/pybigquery/lib/python3.9/site-packages/google/cloud/bigquery/dbapi/cursor.py:165: in execute
    query_parameters = _helpers.to_query_parameters(parameters)
../../miniconda3/envs/pybigquery/lib/python3.9/site-packages/google/cloud/bigquery/dbapi/_helpers.py:168: in to_query_parameters
    return to_query_parameters_dict(parameters)
../../miniconda3/envs/pybigquery/lib/python3.9/site-packages/google/cloud/bigquery/dbapi/_helpers.py:147: in to_query_parameters_dict
    param = scalar_to_query_parameter(value, name=name)
../../miniconda3/envs/pybigquery/lib/python3.9/site-packages/google/cloud/bigquery/dbapi/_helpers.py:51: in scalar_to_query_parameter
    parameter_type = bigquery_scalar_type(value)
../../miniconda3/envs/pybigquery/lib/python3.9/site-packages/google/cloud/bigquery/dbapi/_helpers.py:193: in bigquery_scalar_type
    scalar_object = pyarrow.scalar(value)
E   AttributeError: 'NoneType' object has no attribute 'scalar'
@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery API. label Mar 12, 2021
@tswast tswast added priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. and removed api: bigquery Issues related to the googleapis/python-bigquery API. labels Mar 12, 2021
@tswast tswast changed the title DB-API depends on pyarrow when query parameters are used DB-API depends on pyarrow when decimal query parameters are used Mar 12, 2021
@plamut plamut self-assigned this Mar 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants