Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BigQuery component not working with custom QueryJobConfig #2612

Closed
parthmishra opened this issue Nov 14, 2019 · 0 comments
Closed

BigQuery component not working with custom QueryJobConfig #2612

parthmishra opened this issue Nov 14, 2019 · 0 comments

Comments

@parthmishra
Copy link
Contributor

parthmishra commented Nov 14, 2019

What happened:
If you pass in a dict for job_config, it's never actually converted into a QueryJobConfig type object that is required by the BigQuery client library. I'm also not able to pass in an object of type QueryJobConfig since job_config requires a dict. Because just the dict gets passed to the client.query() call, I end up with the following error:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/local/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/ml/kfp_component/launcher/__main__.py", line 34, in <module>
    main()
  File "/ml/kfp_component/launcher/__main__.py", line 31, in main
    launch(args.file_or_module, args.args)
  File "kfp_component/launcher/launcher.py", line 45, in launch
    return fire.Fire(module, command=args, name=module.__name__)
  File "/usr/local/lib/python2.7/site-packages/fire/core.py", line 127, in Fire
    component_trace = _Fire(component, args, context, name)
  File "/usr/local/lib/python2.7/site-packages/fire/core.py", line 366, in _Fire
    component, remaining_args)
  File "/usr/local/lib/python2.7/site-packages/fire/core.py", line 542, in _CallCallable
    result = fn(*varargs, **kwargs)
  File "kfp_component/google/bigquery/_query.py", line 67, in query
    query_job = client.query(query, job_config, job_id=job_id)
  File "/usr/local/lib/python2.7/site-packages/google/cloud/bigquery/client.py", line 1437, in query
    query_job = job.QueryJob(job_ref, query, client=self, job_config=job_config)
  File "/usr/local/lib/python2.7/site-packages/google/cloud/bigquery/job.py", line 2298, in __init__
    if job_config.use_legacy_sql is None:
AttributeError: 'dict' object has no attribute 'use_legacy_sql'

What did you expect to happen:
The job_config dict should be cast to QueryJobConfig object using .from_api_repr() or allowing a QueryJobConfig object to be passed directly since I don't believe this happens automatically in the client library.

Am I doing this incorrectly?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant