list_tables() returns a maximum of 1000 table names

[TroyHernandez]

length(list_tables(project, dataset, max_results = 1001))
[1] 1000

I have over 3000 tables in the dataset and could really use all of their names.

[craigcitro]

so the BQ API has a limit on how many results it will return.

• for bigrquery, we should definitely handle that paging ourselves -- i'm going to assume that's what you wanted here.
• that said, if you need something right now as a workaround, you could fish out the pagetoken and include it on the next request, and it'd let you pull the next 1000 tables. (and, of course, repeat as necessary)

from past experience, the slick thing to do here is to fetch lazily, which we should make sure we do.

[TroyHernandez]

I'm not sure what bq_get() does, but it looks like it's getting more than just the names. Right now I'm doing this as a workaround:

length(system("bq ls --max_results=1001 project:dataset", intern = TRUE))
[1] 1001

[hadley]

@craigcitro do you know of any big public datasets that we could use for testing here?

[craigcitro]

@hadley I think the easier approach is to expose the param that controls number of items per API call (either as a param or just for tests), and then just set that to 1 and try something with at least 2 items.

[hadley]

Oh, duh, yeah.

[hadley]

Duplicate of #108

[TroyHernandez]

A little late to this... and it was two jobs ago, but IIRC bq_get() for this large number of tables was super slow because it was returning a bunch of additional unnecessary info... just a warning.

[craigcitro]

Good call @TroyHernandez -- just filed #153 for that.