-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bigquery: RowIterator resets token after first page #2694
Comments
Thanks for the report. This is somewhat unusual idiom, as you're specifying a token defined by the query result, but you're executing a new query each time. There's no guarantee that the pagination tokens for one query are valid for another. So if your data is changing at all this is going to yield incorrect results, but for results that yield cached results you're probably getting stable tokens from query to query. If you reach If you want to consume results by page rather than by element, https://pkg.go.dev/google.golang.org/api/iterator?tab=doc has a Pager abstraction that allows you to consume elements by page rather than by element, which is similar to what you've built here. If I've misunderstood, please provide more details. |
Thanks @shollyman for the swift answer. What I'm trying to achieve is 20 items sized pages the frontend will be asking via gRPC so subsequent request would go with a token from the previous one. Maybe I'm doing it all wrong but that's not obvious from the docs. EDIT: |
It's not a particularly important detail, but the bigquery library is still based on REST and JSON, not gRPC. Maybe this example will help: func TestIntegration_PagedQueryIterator(t *testing.T) {
if client == nil {
t.Skip("Integration tests skipped")
}
ctx := context.Background()
// This uses the shortened invocation of asking for an iterator given a query config
// which elides some of the query execution details.
//
// You may want to run the query via q.Run() and retain a reference to the Job ID (string),
// so you can reconstruct an iterator and pager from the JobID each time.
q := client.Query("SELECT * from bigquery-public-data.samples.shakespeare LIMIT 99")
it, err := q.Read(ctx)
if err != nil {
t.Fatalf("couldn't query: %v", err)
}
pageSize := 12
currentToken := ""
totalPages := 0
tokenSet := make(map[string]struct{})
for {
// You could continue to ask for pages from a single pager instance.
// This example builds the pager from an iterator for each page to
// demonstrate managing tokens directly.
pager := iterator.NewPager(it, pageSize, currentToken)
var page [][]Value
nextToken, err := pager.NextPage(&page)
if err != nil {
t.Fatalf("err getting page: %v", err)
}
totalPages = totalPages + 1
tokenSet[nextToken] = struct{}{}
if len(page) != pageSize && nextToken != "" {
// This is a brittle constraint. The backend can choose to limit
// page size due to response size, but we're asking for trivial
// volumes of data. Be aware of this for larger pages.
t.Errorf("got an unexpected smaller page size of %d for token %s", len(page), currentToken)
}
if nextToken == "" {
break
}
currentToken = nextToken
}
wanted := 9
if totalPages != wanted {
t.Errorf("wanted %d pages, got %d", wanted, totalPages)
}
if len(tokenSet) != wanted {
t.Errorf("wanted %d distinct tokens, got %d", wanted, len(tokenSet))
}
} |
Thanks 🙏! Just to clarify, I meant my frontend talks to my backend using gRPC protocol. I did not mean that bigquery does that. I want the end user to request a new page on each subsequent request. |
I'll go ahead and close this issue out now, as there's nothing here indicating a problem with the row iterator mechanism. I hope the example was sufficient for you to build a page-based mechanism. If I've missed something, please comment here or open a followup issue. |
@shollyman The example does not work between rpc calls This was not trivial at all. hope it helps to anyone. |
Client
BigQuery
Environment
GKE
Go Environment
$ go version
$ go env
Code
e.g.
Expected behavior
When calling this function multiple times when the
token
returned from previous call it should return the next page.Actual behavior
It fails on the
Next(&row)
call as iterator.Done errorScreenshots
The text was updated successfully, but these errors were encountered: