Support a query over an external data source / federated data source

[brandondutra]

Bigquery supports running queries over csv files, but I don't see this option in this bigquery api.

See https://cloud.google.com/bigquery/external-data-sources
and
https://cloud.google.com/bigquery/external-data-cloud-storage

I see example for reading csv files into a table, but I would rather not have to manage creating and deleting temporary tables.

[dhermes]

AFAIK this is supported out of the box, but I'm not sure how.

@tswast Do you have any docs to this effect?

[tswast]

No samples yet for this. It's on my todo list.

[dhermes]

@tswast Is there anything you'd like to see from us, e.g. a Client.from_foo() factory constructor that allows specifying scopes for the federated sources? (Or is that necessary?)

[tswast]

Yeah, it is necessary, thus: #3067 The Google Cloud Platform scope covers Cloud Storage, but not Drive or Sheets. (Not sure about Bigtable)

Regarding those Client.from_foo() functions, I'd want us to make sure we pass through all the supported arguments in the google-auth library.

As far is the external data source functionality goes, the Python client libs are missing a bit it seems.

Looks like QueryJob is missing the tableDefinitions parameter.

It also looks like Table is missing external data configuration.

You can create an external data table in the UI and then query it fine, though. Once the external table definition is defined you can query it like any other table.

[brandondutra]

I cannot use the UI. I'm looking to replace Google's Datalab python/bigquery api with this one.

[lukesneeringer]

Hey all,
While triaging today, I deteremined that we have two issues for this, the other being #2372. I am closing this one in favor of that one.

[dhermes]

@lukesneeringer Agree. Both this one and #2372 are BigQuery specific. We should probably have a more generic: "non-default scopes?" issue