Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mobile_twitter_to_rss not working #97

Open
kalaklanar opened this issue Dec 19, 2020 · 5 comments
Open

mobile_twitter_to_rss not working #97

kalaklanar opened this issue Dec 19, 2020 · 5 comments

Comments

@kalaklanar
Copy link

My local install has been running mobile_twitter_to_rss just fine for months, but broke recently (within the last 2 days).

I did some digging and was getting a 400 error until I changed line 28 of the current master to use a full User Agent string:
$browser->agent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:84.0) Gecko/20100101 Firefox/84.0');

I now get a 404, which is a redirect after getting a result in the perl debugger with the $response->content containing "We've detected that JavaScript is disabled in this browser." I'm guessing the page content that LWP was pulling in was non-JS, and that's gone now. I've reached my limit of digging for now. I may dig further later, but I have a question before I try:

Were the old LWP results pulling in a non-JavaScript version from mobile.twitter.com?

If not, this may be heavy lifting and not worth my efforts. (I may try working with nitter if the non-JS was what it was using, and twitter has killed that. but I like how twitrrsme is exactly what I want, and nothing more.)
@EricBrunel
Copy link

What is the "external" symptom of the error you describe? I have a local installation of TwitRSS.me too, which has been working fine until a few days ago. Now, all requests fail with an error "ERR: Can’t screenscrape Twitter". I wanted to open an issue for that, but I wouldn't want to create a duplicate.

@kalaklanar
Copy link
Author

Here is the full error I get when running it on the command line:

Content-type: text/html
Status: 404
Cache-control: max-age=86400
Refresh: 10; url=http://twitrss.me

<html><head></head><body><h2>ERR: Can&#8217;t screenscrape Twitter</h2><p>Redirecting you back to <a href="http://twitrss.me">TwitRSS.me</a> in a few seconds. You might have spelled the username wrong or something</p></body></html>

@chrislombardo
Copy link

chrislombardo commented Dec 25, 2020 •
edited
Loading

Is this the end of the road for twitrss.me? My local installation has been down since 12/16/2020.
"We've detected that JavaScript is disabled in this browser. Please enable JavaScript or switch to a supported browser to continue using twitter.com. You can see a list of supported browsers in our Help Center."
We could probably rebuild this using selenium webdriver with a headless browser to pull the pages for scraping, if there is no quicker solution out there.

@kalaklanar
Copy link
Author

kalaklanar commented Dec 25, 2020 •
edited
Loading

https://github.com/zedeus/nitter is a solution that is already built, but it's a full Twitter interface without java script that has RSS support. It's a cudgel if you just want RSS that links to twitter, but it is what I'm currently using as a workaround to twitrssme breaking with Twitter's update. They have links to hosted versions of it, so you can just make a new feed to one of those servers. If you want to interact with the individual twitter posts (like comment on one), you have to open the link in the RSS post (which displays the post on the hosted Nitter site), then click on the twitter icon to go the original page.

I'm thinking of a script to have the https://github.com/FreshRSS/FreshRSS installation I'm using on my server to change the links to the hosted Nitter site to point to the original Twitter post, but that would be something I'd add for myself, possibly cleaning it up to make a PR for FreshRSS.

https://github.com/RSS-Bridge/rss-bridge may work, but I just found it again, so I haven't set it up.

@chrislombardo
Copy link

Nitter works like a champ. Had to switch to using nitter.dark.fail as the source URL because the main site was overloaded, but it all pairs very nicely with my RSS reader (The Old Reader) with the same feature set I was used to from twitrssme.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kalaklanar @chrislombardo @EricBrunel