Allow custom SQL query for GBQTableDataSet

[simonpicard]

Description

Similar to SQLTableDataSet and SQLQueryDataSet, I would like to have a GBQQueryDataSet where I can specify to query to run on GBQ.

Context

I am building a data pipeline where I increment data every day in GBQ tables, therefore I need to be able to select a portion of the data which has not been processed yet.

Possible Implementation

Create a new class inheriting from GBQTableDataSet with an extra arg sql.
sqlis then passed to pd.read_gbq as is in the _loadfunction.
I actually implemented it that way as a custom dataset for my project.

Possible Alternatives

n/a

Thanks for reading!

[datajoely]

Hi @simonpicard thanks for raising the issue - makes sense and I'll add it to the backlog.

If you need it quickly the best thing to do is to implement it yourself by inheriting the existing class and customising it for your own purposes. There are some instructions on how to do this here.

If you do manage to implement this we would really appreciate a PR if you have the time? In theory it should match the pandas.SQLQueryDataSet implementation + the nifty recent contribution to point to a *.sql file.

[lorenabalan]

Closing this as resolved in #1039

[lorenabalan]

Thank you @simonpicard!!