Add option to re-fetch on every next-page and load more.

[bidoubiwa]

Original comment: #389 (comment)

Problem

On the first search query, we are fetching by default 200 hits that can be changed using paginationTotalHits in the parameters. The reason we do this is because of the pagination widget needs to add the possibility to go to the last page:

MeiliSearch is not able to calculate the correct total number hits. Thus we decided to manually add a limit to be able to provide the button to the last page without creating a huge performance issue. As pagination is very expensive for MeiliSearch. In the background, how more you click on next page, how more MeiliSearch needs to sort a growing number of documents again and again every time you click on next. See extended reasoning.

Algolia has the same issue.

Note: Since we added the cache system, only one request is done per search and pagination do not trigger a new request.

Another problem is that we chose this option just for users using the pagination widget. Which is less modern than the more common infinitePagination with the load more button or the scroll behavior.

Solution

Since we want to promote the infinitePagination behavior we should make the whole fetch 200 hits on each sort a parameter that you can enable. Which by default would be false.

Solution implementation

1. Add a option in the instantmeiliSearchOptions named something like finitePagination that uses the parameter paginationTotalHits. The behavior would be the same as today.
2. The default behavior would be to fetch with a limit equal to the parameter hitsPerPage. On each search, on every pagination, it will only fetch that amount. The offset is equal to page * hitsPerPage
3. Because we still want the load more to appear in case there are more results. The limit should be hitsPerPage + 1
4. Store all results in cache, but this time pagination parameter should be added in cache.

Related to #521