BUG: Scroll API failing on index:export and collection:export

[leonardgable]

Using kourou to dump my collection stored on elastic search the kourou cli fails on big collections.
The command that are failing are:
kourou index:export
and
kourou collection:export

Expected Behavior

I am supposed to be able to dump the whole collection directly into a JSONL format file

Current Behavior

The command kourou collection:export fails while exporting the data to a JSONL format file.
It seems the scroll API closes itself before the whole collection has been retrieved.

Possible Solution

I suspect that the sharding starts to fails on elastic search on big collections.

Steps to Reproduce

Try to dump a collection that has more than 17000 documents insides, and export it to a JSONL file using the kourou collection:export command

Context (Environment)

Trying to extract my data stored using kuzzle framework into jsonL files.

Kuzzle version: 2.18.1
Node.js version: 16.15.1
SDK version: 7.10.1

[Aschen]

Hi @leonardgable

This error indicate an expired scroll search

If the network is too slow to send you an entire page of result, then the current page TTL may expire.

You can use the scrollTTL option to increase the TTL with the kourou collection:export command

[leonardgable]

Hi @Aschen

Indeed it solves the issue. Would be a nice trick to extend the default of scrollTTL.
Is there any particular reason why it is set with such a short expiring term ?

[Aschen]

Hi @Aschen

Indeed it solves the issue. Would be a nice trick to extend the default of scrollTTL. Is there any particular reason why it is set with such a short expiring term ?

No particular reason, it will be extended in the next version 👍 Thanks for rising the issue