Avoid 10K cap on numbers of results

[anjackson]

For some pages, we hit OutbackCDX's default limit=10000 for numbers of results, e.g. https://beta.webarchive.org.uk/wayback/archive/*/https://www.theguardian.com/uk which stops in 2018.

How can we change pywb to use a larger limit?

[anjackson]

Similarly, prefix searches seem to top-out at 100,000? https://beta.webarchive.org.uk/wayback/archive/*?url=http://www.bl.uk/&matchType=prefix

But if I query the https://beta.webarchive.org.uk/wayback/archive/cdx?url=http://www.bl.uk/&matchType=prefix&limit=1000000 I get 145,793?!

[anjackson]

Also, this may be a separate thing, but I notice there's no 'collapsing' happening even when instances are pretty close together (i.e. every snapshot has it's own line with a (1) at the end). Not sure this is set up right?

[ikreymer]

Looks like the limit is 100,000 if no other limit is specified, for both prefix and exact.
Can add a config.yaml setting:
query_limit: 50000 to override. Or support different limits for exact and prefix?

Will look at collapsing, think that's a separate issue.

[anjackson]

That configuration setting would work great!

[ikreymer]

Configuration setting added! Included in webrecorder/pywb:2.4.0-rc1 Docker image.

[anjackson]

Tried this and it didn't appear to work. How should I configure it?

EDIT: I put query_limit: 100000 at the top level of the config. Presumably it does work on any query back end? Or is it only the XmlQuery/OpenSearch API?

[ikreymer]

Yes, it should work for any backend. I've tested smaller limits, but not larger ones.. Just in case, does it work if you set it to be small, eg. 10?

[ikreymer]

Actually, looks like it does not currently work on the XmlQuery backend, as that requires slightly different semantics, not just adding &limit=10000.. will fix soon

[ikreymer]

Oops, didn't mean to close this yet. Re-opening so you can double-check that its working @anjackson

[anjackson]

Looks like it's fixed, based on https://beta.webarchive.org.uk/wayback/archive/*/https://www.theguardian.com/uk right now. Seems a bit slow but I think that might be something wholly separate.