Change getGuestToken to use Twitter-API V1.1 guest/activate for reque…

[arnd-s]

Change getGuestToken to use Twitter-API V1.1 guest/activate for requesting new guest tokens

Instead of searching inside base html page for the guest token, this patch instead uses the Twitter REST API V1.1 to aquire the nessecary guest tokens.

[berenddeschouwer]

This does work for me in brief tests.

[arnd-s]

Have you flushed you cache ? What kind of error do you have ? 403 or 500 Could not parse guest token.
Have you any messages inside webservers error_log ?

[berenddeschouwer]

Have you flushed you cache ? What kind of error do you have ? 403 or 500 Could not parse guest token. Have you any messages inside webservers error_log ?

Flushed both rss-bridge and php

The previous logs are pointless, because I was running my own fork to attempt to fix the previous 403/500 problems. So they're not stock rss-bridge logs.

I got 500s (could not parse guest token), which then if you force refresh gets 403-s. The 500 is a rss-bridge error if twitter 200s without a guest token. 403 is straight from Twitter, from memory.

A full test would need at least 3 more hours, for the cookies to expire.

[arnd-s]

On my side the patch is working fine for some hours now. I dropped the cache multiple times and had no issue so far.

Maybe add some debug to getGuestToken function:

error_log("ApiKey: ".$apiKey);
error_log("GuestToken: ".$guestToken);

You can also test the API Call via curl. Just paste the Api Key from your error log, if you added the debug

curl -H"authorization: Bearer $apiKey" -d"" -v -v https://api.twitter.com/1.1/guest/activate.json

Should return something like

{"guest_token":"1479486489708974084"}

If this works, then the theory is at least okay.

[berenddeschouwer]

I'm quite happy that this PR fixes the main bug.

There's a secondary bug where rss-bridge blindly assumed the guest token timeout provided by twitter was accurate, and that wasn't always the case. That's where the 403s come in. You'd need to run for a few hours without deleting the cache for it to trigger.

NOTE: that's a secondary bug. The guest token (this PR) needs to get fixed first. If the new API works correctly, rss-bridge might never encounter the second bug.

[arnd-s]

Yeah, you are right. The 403 is another issue, which we need to solve separately.

My first thinking is to drop cached tokens on a 403 if they are older then X seconds... I will look into it an make a separate PR if i'm able to solve this.

[berenddeschouwer]

I solved it in the old API exactly that way. On 403, get new tokens.

It worked for about 24 hours, and then this happened :)

I also reset the guest token timeout if we re-used it successfully, but I couldn't test if that helped or hurt before twitter changed api versions.

And I changed the 500 to a 503 (http temporary error) which makes more sense. This helped for my rss reader (freshrss), but I don't know if that's good or bad for other rss readers.

[berenddeschouwer]

I've seen the 403s again, so I've put my "if 403, re-get tokens" code back in.

I'm trying to setup a PR now for that. Two notes:

• my PR would require this PR first. It doesn't work without the API v1.1 changes
• for now, I've got lots of Debug::log, including page contents, so I don't recommend anyone run my code in production. I need some statistics on 403/500.

[arnd-s]

This PR is currently missing a check, if the API call was successful or not. I will later today add this to the PR.

[berenddeschouwer]

Forked your branch and created PR #2416

[doowruc]

I have been running this version for a couple of days from an Azure Web App and have had no 500 or 403

[em92]

Temporarly deployed on my public instance (https://feed.eugenemolotov.ru)

[em92]

Also applied this patch to invalidate previous guest tokens.

diff --git a/bridges/TwitterBridge.php b/bridges/TwitterBridge.php
index a14e506..e3b7585 100644
--- a/bridges/TwitterBridge.php
+++ b/bridges/TwitterBridge.php
@@ -539,13 +539,13 @@ EOD;
                $cacheFac2->setWorkingDir(PATH_LIB_CACHES);
                $gt_cache = $cacheFac->create(Configuration::getConfig('cache', 'type'));
                $gt_cache->setScope(get_called_class());
-               $gt_cache->setKey(array('guest_token'));
+               $gt_cache->setKey(array('guest_token2'));
                $guestTokenUses = $gt_cache->loadData();
 

@berenddeschouwer, is that solution ok for you instead of your PR?

[berenddeschouwer]

Also applied this patch to invalidate previous guest tokens.

diff --git a/bridges/TwitterBridge.php b/bridges/TwitterBridge.php
index a14e506..e3b7585 100644
--- a/bridges/TwitterBridge.php
+++ b/bridges/TwitterBridge.php
@@ -539,13 +539,13 @@ EOD;
                $cacheFac2->setWorkingDir(PATH_LIB_CACHES);
                $gt_cache = $cacheFac->create(Configuration::getConfig('cache', 'type'));
                $gt_cache->setScope(get_called_class());
-               $gt_cache->setKey(array('guest_token'));
+               $gt_cache->setKey(array('guest_token2'));
                $guestTokenUses = $gt_cache->loadData();
 

@berenddeschouwer, is that solution ok for you instead of your PR?

Are you trying to invalidate it between running instances to test if source code changes work? In that case I suspect you need to remove the files in cache/TwitterBridge/ or restart memcached instead.

If you're trying to invalidate it between every call to getApiKey(), you're better off not caching it at all.

[em92]

gj, @arnd-s!
How about setting you to maintainers of this bridge?

[arnd-s]

@em92 Would be okay for me.