Is it possible to use "next_token"?

[myrainbowandsky]

I tried but does not work. Does that mean I should restart the whole search?

[SamHames]

If you're using the command line interface, yes, you'll need to restart the search. If you're seeing repeated API errors in the twarc.log file (such as 503 errors), it might be more stable for you to turn off context annotations (--no-context-annotations) and turn down the maximum results per page (--max-results (try 50/100)).

We have thought about resuming in the past (see #589), but it's quite a bit of work to do well and I don't think there's a lot of motivation around here to do that work given the state of Twitter as a platform right now.

[igorbrigadir]

Just for reference, you can add a next_token yourself, if you're using Twarc as a library, https://twarc-project.readthedocs.io/en/latest/api/library/ the only thing missing is the command line commands and appropriate appending to existing files etc. So you'll have to read your own next_token from someplace, and make sure you're writing the output yourself: https://twarc-project.readthedocs.io/en/latest/api/client2/#twarc.client2.Twarc2.search_all

[myrainbowandsky]

Thank you @igorbrigadir @SamHames . For command line, I think i can check the create_at and set the time window, which could be an alternative.