Add Twitter::SearchResults methods for handling next results

[KentonWhite]

Twitter API v1.1 pages the search results. The information for the next page of search results is contained in the search_metadata field as a query string for the URL. This makes it difficult to use with the Twitter gem, which stores the query parameters as an options hash.

Added 2 methods to Twitter::SearchResults: next_results? and next_results

next_results? returns true if next_results if defined, false otherwise

next_results returns a hash of query parameters that can be merged into the original search options hash. Using the new hash then returns the next page of results.

Example usage:

search_results = twitter.search(query, opts)
while search_results.next_page? 
  opts.merge! search_results.next_page
  search_results = twitter.search(query, opts)
end

For backwards compatibility with v1 API, aliased next_results? to next_page? and next_results to next_page

[sferik]

Thanks for the patch. My biggest concern is that Twitter::SearchResults#next_page returns a Hash that contains a variety of keys, many of which are unrelated to the next page.

When I needed to fetch all the results for a particular query, I wrote code like this:

def collect_with_max_id(collection=[], max_id=nil, &block)
  tweets = yield(max_id)
  return collection if tweets.nil?
  collection += tweets
  tweets.empty? ? collection.flatten : collect_with_max_id(collection, tweets.last.id - 1, &block)
end

query = "ruby on rails"
opts = {:count => 100} # specify whatever opts you want here
tweets = collect_with_max_id do |max_id|
  opts[:max_id] = max_id unless max_id.nil?
  Twitter.search(query, opts).results
end

Is there a compelling reason why your solution is better than this?

[KentonWhite]

I agree with your concern that the hash contains keys that might be unrelated to the next page. However one doesn't need to use all of the keys. If all you are interested in is the :max_id parameter then just this parameter can be extracted from the hash and your options updated as per your example.

Would the interface be better if we return just the :max_id parameter and discarded the rest? I toyed with this idea but rejected it as potentially over restrictive. Do you have any thoughts?

The example you give is very good but has 2 drawbacks over using the next_results hash as returned by the Twitter API.

1. Efficiency Each search must make one additional call to the Twitter API to check if there are any additional entries. In the worst case this will result in the code making twice as many calls to Twitter than is necessary. This slows down execution and wastes rate limited requests. Using next_requests? does not require a separate call to Twitter to determine if there are more search results to fetch.
2. Assumed Ordering The method of looking at the last tweet's ID to determine the next : max_id assumes that tweets returned from Twitter are returned ordered descending by their ID. No where in the V1.1 API documentation does Twitter guarantee the ordering of their Tweets. While the ordering is plausible it may be just a coincidence at the moment and could change in the future. Should the ordering be something other than descending by ID it could result in duplicate tweets (e.g. if the ordering is random) or an infinite loop (e.g. if the ordering is reversed to ascending by ID). On the other hand, Twitter guarantees that the next_result field will contain the query for retrieving the next page of results. Using :max_id returned from this field does not require any assumptions about the order of tweets.

[sferik]

@KentonWhite:

I agree with your concern that the hash contains keys that might be unrelated to the next page. However one doesn't need to use all of the keys. If all you are interested in is the :max_id parameter then just this parameter can be extracted from the hash and your options updated as per your example.

Would the interface be better if we return just the :max_id parameter and discarded the rest? I toyed with this idea but rejected it as potentially over restrictive. Do you have any thoughts?

There is already a Twitter::SearchResults#max_id method. I'm not sure what benefit you get pulling it out of a Hash? It seems simple enough to say:

results = Twitter.search(query, opts)

results.max_id
# versus
results.next_results[:max_id]

Efficiency Each search must make one additional call to the Twitter API to check if there are any additional entries. In the worst case this will result in the code making twice as many calls to Twitter than is necessary. This slows down execution and wastes rate limited requests. Using next_requests? does not require a separate call to Twitter to determine if there are more search results to fetch.

To be clear, I have no objection to adding Twitter::SearchResults#next_results?. If you separated this into two pull requests (one that implements Twitter::SearchResults#next_results? and one that implements Twitter::SearchResults#next_results), I'd be happy to immediately merge the former while we continue discussing the latter.

Assumed Ordering The method of looking at the last tweet's ID to determine the next :max_id assumes that tweets returned from Twitter are returned ordered descending by their ID. No where in the V1.1 API documentation does Twitter guarantee the ordering of their Tweets. While the ordering is plausible it may be just a coincidence at the moment and could change in the future. Should the ordering be something other than descending by ID it could result in duplicate tweets (e.g. if the ordering is random) or an infinite loop (e.g. if the ordering is reversed to ascending by ID). On the other hand, Twitter guarantees that the next_result field will contain the query for retrieving the next page of results. Using :max_id returned from this field does not require any assumptions about the order of tweets.

I'm just following the instructions in Twitter's guide to Working with Timelines. Specifically, they recommend:

Subtract 1 from the lowest Tweet ID returned from the previous request and use this for the value of max_id.

That said, your point is reasonable. Someone asked essentially the same question on the developer forums a year ago. This was the official reply, verbatim:

If you're curious about how IDs are generated these days, check out http://engineering.twitter.com/2010/06/announcing-snowflake.html

They are mostly sequential, but they've got a bit more complexity to them. In the vast majority of cases, doing a -1 or +1 trick on a status is sufficient. If you want to be sure, you can always just use a preserved ID and tolerate duplicates in your result.

I'm not saying "no". I just want to make sure this is necessary.

[KentonWhite]

To be clear, I have no objection to adding Twitter::SearchResults#next_results?. If you separated this into two pull requests (one that implements Twitter::SearchResults#next_results? and one that implements Twitter::SearchResults#next_results), I'd be happy to immediately merge the former while we continue discussing the latter.

Closing this pull request and splitting into 2 separate pull requests. Will continue the discussion on Twitter::SearchResults#next_results on the new pull request.

[KentonWhite]

There is already a Twitter::SearchResults#max_id method. I'm not sure what benefit you get pulling it out of a Hash? It seems simple enough to say:

results = Twitter.search(query, opts)

results.max_id
# versus
results.next_results[:max_id]

Yes there is such a method, and it pulls the :max_id from the :search_metadata structure. Unfortunately this is not the correct :max_id for making a request to the next page. It is instead the :max_id that was sent with the original request. From the example in the Twitter v1.1 documentation:

https://api.twitter.com/1.1/search/tweets.json?q=%23freebandnames&since_id=24012619984051000&max_id=250126199840518145&result_type=mixed&count=4

"search_metadata": {
  "max_id": 250126199840518145,
  "since_id": 24012619984051000,
  "refresh_url": "?since_id=250126199840518145&q=%23freebandnames&result_type=mixed&include_entities=1",
  "next_results": "?max_id=249279667666817023&q=%23freebandnames&count=4&include_entities=1&result_type=mixed",
  "count": 4,
  "completed_in": 0.035,
  "since_id_str": "24012619984051000",
  "query": "%23freebandnames",
  "max_id_str": "250126199840518145"
}

The only way to get the correct :max_id for the next page is to

1. Find the smallest ID in the current results and subtract 1.
2. Parse the :max_id from the :next_results string.

Looking again at the documentation example, Twitter is just subtracting 1 from the ID of the last returned tweet. So the 2 methods are equivalent.

I guess the question is do we want a convenience method for getting the :max_id to use in paging through search results? If the answer is yes, how should this convenience method work? Call itself something different, e.g. next_page_id or return aHash of values for the developer as I implemented with next_results.

Both have obvious advantages and disadvantages. Providing a Hash of values with a next_results method mirrors the Twitter search result structure, but is confusing since users have to pick the correct key from the Hash. On the other hand a method next_page_id is simple but now introduces a field that is not in the original returned structure, which can also be confusing.

Perhaps the best solution is to just use Twitter.search(query, opts).results.last.id - 1. Even though verbose it's intention is quite explicit.

[sferik]

I need some more time to think about the best thing to do here.

In the mean time, I've merged #366 into master.

[paracycle]

I recently also felt the need for something like next_results and I had come up with:

def next_page_params(next_page)
  p = URI.decode_www_form next_page[1..-1]
  params = Hash[p]
  params.keys.each do |key|
    params[(key.to_sym rescue key) || key] = params.delete(key)
  end
  q = params.delete(:q)
  return  q, params
end

to feed the next call to Twitter.search. @KentonWhite has a much cleaner solution that I would have used.

I agree with @KentonWhite that we should let the application developer decide if they want to use the parameters that Twitter returns for making a call for the next page, or to come up with other solutions.

[sferik]

Okay, I am convinced that this is necessary.

[sferik]

You may be interested in this recent refactoring: fdd8388