Cloud Run fails to serve database files larger than 32MB

[simonw]

https://cloud.google.com/run/quotas lists the maximum response size as 32MB.

I spotted a bug where attempting to download a database file larger than that from a Cloud Run deployment (in this case it was https://github-to-sqlite.dogsheep.net/github.db after I accidentally increased the size of that database) returned a 500 error because of this.

[simonw]

Proposed solution: on Cloud Run don't show the "download database" link if the database file is larger than 32MB.

I can do this with a new config setting, max_db_mb, which is automatically set by the publish cloudrun command.

This is consistent with the existing max_csv_mb setting: https://datasette.readthedocs.io/en/stable/config.html#max-csv-mb

I should set max_csv_mb to 32MB on Cloud Run deploys as well.

[simonw]

https://cloud.google.com/blog/products/serverless/cloud-run-now-supports-http-grpc-server-streaming indicates this limit should no longer apply:

With this addition, Cloud Run can now ... Send responses larger than the previous 32 MB limit

But I'm still getting errors from Cloud Run attempting to download .db files larger than 32 MB.

I filed a question in their issue tracker about that here: https://issuetracker.google.com/issues/173038375

[simonw]

I asked about this on Twitter - https://twitter.com/steren/status/1334281184965140483

You simply need to send the Transfer-Encoding: chunked header.

[simonw]

This works!

/tmp % wget 'https://covid-19.datasettes.com/covid.db'
--2020-12-02 16:28:02--  https://covid-19.datasettes.com/covid.db
Resolving covid-19.datasettes.com (covid-19.datasettes.com)... 172.217.5.83
Connecting to covid-19.datasettes.com (covid-19.datasettes.com)|172.217.5.83|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/octet-stream]
Saving to: ‘covid.db’

covid.db                [              <=>   ] 306.42M  3.27MB/s    in 98s     

2020-12-02 16:29:40 (3.13 MB/s) - ‘covid.db’ saved [321306624]