Merger is slow to start when there are many one-block files

[matthewdarwin]

Merger is slow to start when there are many one-block files piled up in s3 bucket. (reading list from s3 is not fast)

In older implementations of merger it would read the first 2000 files (approximately) and take action from that. There is no need to read all the files.

[sduchesneau]

99f16fc
and
fcb32d7

should fix this.

pushed in streamingfast/firehose-ethereum@59e9790

[sduchesneau]

Also, your specific problem is probably related to streamingfast/go-ethereum#1
if the one-block-files have a libnum stuck way in the past, it won't progress until that libnum moves up.

The workaround for the geth issue should get you going, and those two commits linked above should also improve performance.

[matthewdarwin]

Latest merger seems fine with lots of one-block files.