TwitterBridge: Could not parse guest token

[triatic]

I'm seeing an increasing number of these in my twitter bridge feeds:

Error message: `Could not parse guest token`

It happens randomly and doesn't seem to be specific to any one of my twitter feeds.

PHP error:

[15-Dec-2021 11:48:03 UTC] Exception: Could not parse guest token in /home/ubuntu/rss-bridge/lib/error.php:24
Stack trace:
#0 /home/ubuntu/rss-bridge/lib/error.php(42): returnError()
#1 /home/ubuntu/rss-bridge/bridges/TwitterBridge.php(570): returnServerError()
#2 /home/ubuntu/rss-bridge/bridges/TwitterBridge.php(548): TwitterBridge->getGuestToken()
#3 /home/ubuntu/rss-bridge/bridges/TwitterBridge.php(576): TwitterBridge->getApiKey()
#4 /home/ubuntu/rss-bridge/bridges/TwitterBridge.php(586): TwitterBridge->getApiContents()
#5 /home/ubuntu/rss-bridge/bridges/TwitterBridge.php(221): TwitterBridge->getRestId()
#6 /home/ubuntu/rss-bridge/bridges/TwitterBridge.php(241): TwitterBridge->getApiURI()
#7 /home/ubuntu/rss-bridge/actions/DisplayAction.php(135): TwitterBridge->collectData()
#8 /home/ubuntu/rss-bridge/index.php(38): DisplayAction->execute()
#9 {main}

[pcause]

seeing the same. this has been previously working. I am also using http not https for the requests to rss-bridge since all the requests are from the same machine. if i use a browser on that system and use the same query it works. the software making the request is written in php and running php8 and php-fpm under nginx

[em92]

How to reproduce this?
This is working fine on my side.
https://feed.eugenemolotov.ru/?action=display&bridge=Twitter&context=By+username&u=jack&format=Html

[triatic]

It's intermittent.

[everdred]

This is also happening to me intermittently since 12/9 for Twitter user "pants", after months of working without issue.

[triatic]

It likely doesn't make any difference, but just for reference, I am checking the Twitter feeds with FreshRSS running on the same machine as RSS-bridge. PHP 8.0 FPM via Apache.

I have 10 Twitter feeds, each checked once every 20 minutes. On average the error occurs roughly once per hour.

[StrangeRanger]

I'm having this same problem.

[Wed Dec 15 00:13:43.707723 2021] [php7:notice] [pid 140133] [client 172.70.214.53:56452] Exception: Could not parse guest token in /var/www/rss-bridge.randomserver.xyz/lib/error.php:24
Stack trace:
#0 /var/www/rss-bridge.randomserver.xyz/lib/error.php(42): returnError()
#1 /var/www/rss-bridge.randomserver.xyz/bridges/TwitterBridge.php(570): returnServerError()
#2 /var/www/rss-bridge.randomserver.xyz/bridges/TwitterBridge.php(548): TwitterBridge->getGuestToken()
#3 /var/www/rss-bridge.randomserver.xyz/bridges/TwitterBridge.php(576): TwitterBridge->getApiKey()
#4 /var/www/rss-bridge.randomserver.xyz/bridges/TwitterBridge.php(241): TwitterBridge->getApiContents()
#5 /var/www/rss-bridge.randomserver.xyz/actions/DisplayAction.php(135): TwitterBridge->collectData()
#6 /var/www/rss-bridge.randomserver.xyz/index.php(38): DisplayAction->execute()
#7 {main}

I'm using a discord bot called MonitoRSS, compared to @triatic's FreshRSS. This is an example output from the bot when retrieving something from my instance of RSS-Bridge: Bridge returned error 500! (18976) | Wed, 15 December 2021, 1:28 AM PST. Similarly, it's occurring intermittently.

[triatic]

I have created a pull request but I don't think it will close this issue. It should reduce the error frequency though.

[triatic]

It would be useful for people to experiment with higher values of GUEST_TOKEN_USES and GUEST_TOKEN_EXPIRY in TwitterBridge.php

I know I can do 100 hits every three hours before I get errors, but it might be that one or the other of these limits can be further increased still. We want to fetch guest tokens as infrequently as possible.

[pcause]

If I run a second instance of rss-bridge on my server are the guest tokens separate or are they tied to an IP address or something about the machine making the requests? This could be a work around.

[triatic]

I don't know exactly, but tweaking GUEST_TOKEN_USES and GUEST_TOKEN_EXPIRY to request new guest tokens as infrequently as possible is a good idea. We could use this thread to determine the maximum safe value for each.

[arnd-s]

I created PR #2396 which will continue to use old/cached guesttokens until we successfully got a new one.

[berenddeschouwer]

I've been trying to treat either 403 or 500 as EAGAIN. Here's why:

• Sometimes I receive 403 before the guest token is supposed to expire
• Sometimes the expected document.cookie doesn't exist in the document body (I never receive a token in the headers)

If we receive a 403, getting a new guest token and refreshing seems to always work.

If we receive a 500, there's no cookie in the document body -- everything else is there -- and a refresh usually gets a valid guest token, which we can then use to retrieve the contents.

When retrieving the guest token, it's always at the very end of the document. The document is just sometimes short and missing the last line.

So I've got code that loops 3 times with a usleep, and then returns error 503 instead of the default 500 to tell the RSS client to try again later. Assuming the clients are well written, that should be OK. Of course, 503 only matters if error output is http.

I've had to rewrite that code a few times, as I get a bit more information from Twitter every time, and I need to run it for a few days to be confident it's fixed -- or at least better.

[linoth]

Error rate has been extremely elevated today, so if anyone needed to test against this intermittent issue, it would be a good time to gather your data. Pretty sure I've gotten more failed feed retrievals than successful.

I wandered into Twitter's API forum, and there's multiple topics on similar issues, i.e. unable to retrieve tokens, invalid tokens, etc. This post in particular had some findings on token lifespans.

https://twittercommunity.com/t/unable-to-obtain-new-access-token-by-using-refresh-token/164123/7

tl;dr it seems that if you request a new token before yours expires, you're okay. Otherwise, you're not. Given the nature of RSS-Bridge, I don't know that this finding is particularly useful, beyond "yep, something's broken."

Twitter's API status page shows "degraded performance" for the RESTful API. It also seems they don't really use issue history at all. If I had to guess, all of this might be a load issue that Twitter either hasn't addressed, or is expecting to pass on its own, i.e. they're not going to address it.

[triatic]

We may need to start thinking about adding Twitter API v2 support:

https://developer.twitter.com/en/docs/twitter-api/tweets/timelines/introduction
https://developer.twitter.com/en/docs/twitter-api/tweets/search/introduction

[arnd-s]

Seems like they have changed the API now. Guest Tokens are now aquired through a post request to https://api.twitter.com/1.1/guest/activate.json

It's mandatory to have a authorization header to be allowed for making the request:
authorization: Bearer AAAAAAAAAAAAAAAA....

If i use a header which i captured by visiting the website via my browser, i was succesfully able to request a guest token.

So next step would be, to figure out, how to get a authorization token.

[arnd-s]

I create a new PR #2414 which should finaly solve the issues.

[em92]

Closing as resolved