Remote externalities download slows down at the end

[ggwpez]

Somehow the remote externalities get slower at the end. This was determined to not be related to RPC rate limits but seems to be a problem of the externalities.
Our suspicion os that it is related to the size of the KV pairs.
A debug flag like --measure-dl-timing output.json could be added to measure and investigate the KV timings later on.

Example: normally around 12-14 kV/s, it slows down to 200-500 V/s. This was also noticed by @liamaharon.

(Marking as mentor since adding the flag should be simple)

[eagr]

I noticed the related code is deprecated in favor of the standalone try-runtime-cli. Do you still want to do this here? @ggwpez @liamaharon

[liamaharon]

I noticed the related code is deprecated in favor of the standalone try-runtime-cli. Do you still want to do this here? @ggwpez @liamaharon

This issue originates in remote-externalities, which try-runtime-cli depends on.

Same crate you are adding the parallel scraping to :)

[eagr]

Yeah, that part of the change definitely goes this repo. I mean the cli flag part. Do you still want to do that here? :))

[liamaharon]

Yeah, that part of the change definitely goes this repo. I mean the cli flag part. Do you still want to do that here? :))

Seems it needs an issue in both repos. One to add the feature to remote-ext, and another to use it in try-runtime-cli