Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(bigquery): improve RowIterator docs and out of process pagination with Storage API #8419

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions bigquery/bigquery.go
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,8 @@ func NewClient(ctx context.Context, projectID string, opts ...option.ClientOptio

// EnableStorageReadClient sets up Storage API connection to be used when fetching
// large datasets from tables, jobs or queries.
// Currently out of pagination methods like PageInfo().Token and RowIterator.StartIndex
// are not supported when the Storage API is enabled.
// Calling this method twice will return an error.
func (c *Client) EnableStorageReadClient(ctx context.Context, opts ...option.ClientOption) error {
if c.isStorageReadAvailable() {
Expand Down
4 changes: 3 additions & 1 deletion bigquery/doc.go
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,9 @@ To retrieve the job's results from the ID, first look up the Job:
}

Use the Job.Read method to obtain an iterator, and loop over the rows.
Query.Read is just a convenience method that combines Query.Run and Job.Read.
Calling Query.Read is preferred for queries with a relatively small result set,
as it will call BigQuery jobs.query API for a optimized query path. If the query
doesn't meet that criteria, the method will just combine Query.Run and Job.Read.

it, err = job.Read(ctx)
if err != nil {
Expand Down
16 changes: 13 additions & 3 deletions bigquery/iterator.go
Original file line number Diff line number Diff line change
Expand Up @@ -51,13 +51,21 @@ type RowIterator struct {
pf pageFetcher

// StartIndex can be set before the first call to Next. If PageInfo().Token
// is also set, StartIndex is ignored.
// is also set, StartIndex is ignored. If Storage API is enabled,
// StartIndex is also ignored because is not supported. IsAccelerated()
// method can be called to check if Storage API is enabled for the RowIterator.
StartIndex uint64

// The schema of the table. Available after the first call to Next.
// The schema of the table.
// In some scenarios it will only be available after the first
// call to Next(), like when a call to Query.Read uses
// the jobs.query API for an optimized query path.
Schema Schema

// The total number of rows in the result. Available after the first call to Next.
// The total number of rows in the result.
// In some scenarios it will only be available after the first
// call to Next(), like when a call to Query.Read uses
// the jobs.query API for an optimized query path.
// May be zero just after rows were inserted.
TotalRows uint64

Expand Down Expand Up @@ -169,6 +177,8 @@ func isStructPtr(x interface{}) bool {
}

// PageInfo supports pagination. See the google.golang.org/api/iterator package for details.
// Currently pagination is not supported when the Storage API is enabled. IsAccelerated()
// method can be called to check if Storage API is enabled for the RowIterator.
func (it *RowIterator) PageInfo() *iterator.PageInfo { return it.pageInfo }

func (it *RowIterator) fetch(pageSize int, pageToken string) (string, error) {
Expand Down
Loading