Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support use of fields, and use it to minimize data transfer for some paging/polling operations. #153

Closed
craigcitro opened this issue Apr 22, 2017 · 3 comments
Labels
api 🕸️ feature a feature request or enhancement

Comments

@craigcitro
Copy link
Collaborator

(This is an offshoot of #58.)

List operations in the BQ API generally return all information about each object by default -- for trying to examine lists of objects, this is great. However, if we're (say) polling for job completion, or looking for a specific table/dataset, it's often wasteful to pull all the data every time. (In particular, it's annoying for manual investigation/debugging.)

IWBN to use fields (which is a global query param in the API) to reduce the set of fields we're pulling: docs.

@hadley hadley added the feature a feature request or enhancement label Mar 28, 2018
@hadley
Copy link
Member

hadley commented Mar 28, 2018

Is this something I should systematically review for every function? Or are there a couple of main offenders? (e.g. wait_for())

hadley added a commit that referenced this issue Apr 11, 2018
Closes #150. Closes #153.
@hadley hadley closed this as completed in fd8108a Apr 12, 2018
@jennybc
Copy link
Collaborator

jennybc commented Apr 12, 2018

I am also vexed by how to handle fields. In googledrive, we ultimately decided to not handle them, i.e. to use the same fields everywhere (I think the default set?), until that proved to be problematic. So far, so good.

However in googlesheets4, I must handle fields, so am thinking about it once again.

@hadley You closed this with a really huge commit. Did you do anything clever with fields that I should study?

@hadley
Copy link
Member

hadley commented Apr 12, 2018

I don't think there will be any take aways for google sheets here because the way in which I need to use fields is quite different. For example, most bigquery operations take place by inserting a job into a queue, and then regularly polling until it's done. I'm now using fields so that when I poll I just get whether or not it's complete, not a bunch of other information that I don't use.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api 🕸️ feature a feature request or enhancement
Projects
None yet
Development

No branches or pull requests

3 participants