[TwitterBridge] Improve timeline processing for username mode

[fivefilters]

Currently when returning results for a twitter username ('By username' mode in RSS Bridge), quoted tweets don't appear to be handled very well. Here's an example from Glenn Greenwald's Twitter timeline:

This appears as two tweets in the RSS Bridge output, which suggests he retweeted the main tweet and the inner quoted tweet separately:

The problem is also there in a regular quoted tweet by the same user (the quoted tweet appears as a separate entry in the list of results).

I had a look at Nitter's parser and noticed they use the timeline property of the global data object to pick out tweets, which seems to avoid this problem. So this change does something similar when in username mode.

It might be worth doing the same for the other modes, but I haven't tested in those modes much, so for now it defaults to the previous behaviour if not in 'By username' mode.

[fivefilters]

Also noticed that enabling 'no retweets' often produced zero results if the account had retweeted a lot after posting something original. Those original tweets could easily be missed by an RSS reader.

This latest change switces to the search endpoint when 'no retweets' is enabled, resulting in the following query:

from:username exclude:retweets

I also noticed the 'no replies' checkbox has no effect at the moment (it's not referred to again in the code). This change appends 'exclude:replies' if it's enabled, e.g.

from:username exclude:retweets exclude:replies

[em92]

Hi, @fivefilters ! Sorry for the long delay.
In general I could not understand the problem that this PR solves.

[em92]

What I see? Glen retweeted Jake's tweet. Jake's tweet appears in feed, in author Glen was added as a "retweeter". And I don't find anything wrong here.

[fivefilters]

Hi @em92, no problem. Will try to explain a little better.

What I see? Glen retweeted Jake's tweet. Jake's tweet appears in feed, in author Glen was added as a "retweeter". And I don't find anything wrong here.

There's nothing wrong there. Glenn did retweet Jake's tweet. But notice that Jake's tweet ends with a link to another tweet, that's the tweet below it in my screenshot (by Acyn Torabi). The problem is that that tweet (by Acyn) was not retweeted by Glenn (RSS Bridge shows it as retweeted: "RT: @ggreenwald"). So that tweet by Acyn Torabi, shouldn't be returned.

It's in the JSON results that Twitter returns becuase Jake's tweet, the one Glenn retweeted, referred to it (Jake's tweet is what Twitter calls a "quote tweet" - a tweet quoting another tweet). But RSS Bridge shouldn't be returning it as something Glenn retweeted. This is handled correctly by Nitter, but not by RSS Bridge.

When I looked into it, I saw that the large JSON object that both RSS Bridge and Nitter use to return tweets, has two tweet sections. The large tweet section that RSS Bridge uses to pull tweets from ($data->globalObjects->tweets) contains not just the desired tweets, but all connected tweets. That's why Acyn's tweet appears here, even though it wasn't retweeted by Glenn.

There's another section in the same JSON object which lists the tweets that should appear for the user ($data->timeline->instructions[0]->addEntries->entries), this is what Nitter uses to determine which tweets to display, and what the modified code I submitted does for RSS-Bridge too.

If you find a Twitter user that has retweeted a quote tweet, I think the problem will be easier to see. If I come across one, I'll post another comment with another example.

[em92]

Thanks for explanation @fivefilters!