Recover from stream failure with backfill

[edsu]

This came up when addressing #462 and is worth recording as an enhancement idea. To guarantee that data is not lost during streaming it ought to be possible to keep track of the last tweet id and then use it to fetch data that was missed.

1. Record the ID of the last tweet received before the interruption.
2. Record the ID of the first tweet received after the interruption.
3. Spawn a new thread that uses search to read in tweets with IDs between the “before” and “after” IDs.

One little wrinkle here is that the search query that is used will need to be constructed on the fly using the current stream rules. There is also a question of where to write the additional data.

[igorbrigadir]

Oh, i missed this earlier - https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/recovery-and-redundancy-features this should make it easier.

Announcement here: https://twittercommunity.com/t/new-reliability-and-recovery-functionality-available-for-twitter-api-v2-streaming-endpoints/156244

[igorbrigadir]

#549 added the extra parameter to the client2 already, so we need the part that takes a command line parameter and or checks the output file and resumes. I'm inclined to say we should always grab the max replay duration, everything else can deal with duplicates downstream. But something smarter might be good.