Is there a limit to the number of rows retrieved with DELPHI API?

[nickreich]

I am trying to retrieve a lot of data from the DELPHI API, but am having trouble getting all the data I am requesting. Is there a limit that I am running into? For example, I run

source("https://raw.githubusercontent.com/cmu-delphi/delphi-epidata/master/src/client/delphi_epidata.R")
res <- Epidata$fluview(regions = list("nat", "hhs1", "hhs2", "hhs3", "hhs4", "hhs5", "hhs6", "hhs7", "hhs8", "hhs9", "hhs10"), 
                       epiweeks = list(Epidata$range(199740, 201653)),
                       issues = list(Epidata$range(199740, 201653)))
df <- do.call(rbind, lapply(res$epidata, rbind)) 

And I end up with data from only regions "nat" and "hhs1" and only until epiweek 200450. The total number of rows in the resulting dataframe is 3650.

[undefx]

Yes, the limit is 3650 rows per request as you noted - sorry that's not in the documentation. To get all the data, you'll have to query the regions individually, or use smaller batches of epiweeks.

I'll leave this issue open until I have a chance to make a note of this in the documentation.

(BTW, the limit is implemented here.)

[sgratzl]

the new endpoints of the server (e.g., /api.php?endpoint=covidcast -> /covidcast/ is streaming the results back to the client. Thus, we could increase the limit to right now 1M rows.

see also https://github.com/cmu-delphi/delphi-epidata/blob/main/src/server/_config.py#L9