Cloudflare hates RSS

[anthonyryan1]

This isn't a Commafeed bug as much as an external, internet-wide problem. I thought I'd open an issue for brainstorming.

More and more blogs are throwing up Cloudflare in front of them. And Cloudflare is blocking Commafeed from fetching the RSS feeds. The feed fetcher is a bot, so it makes sense. But it's still causing more and more blogs to go offline.

Do you think registering Commafeed as a bot with Cloudflare would help? We've got no hope of stopping more websites from piling up behind Cloudflare.

All ideas welcome.

[Athou]

Hi @anthonyryan1
There was an issue some time ago with cloudflare (see #1138). I used the form to register CommaFeed as a verified bot but got no response, and they didn't add CommaFeed to the list of verified bots. However, the feeds started working a short time after that so I didn't give it more thought.

I can try to submit CommaFeed again, it seems the procedure changed a little bit (it was a google form before).

Do you have an example of a feed that is blocked by cloudflare?

[jkrafczyk]

Do you have an example of a feed that is blocked by cloudflare?

Science.org seems to use Cloudflare, and their RSS feeds get 403 responses when requested via Commafeed or curl.
Example feed URL: https://www.science.org/digital-feed/pipeline

[anthonyryan1]

This issue came up for discussion on Hacker News today: https://news.ycombinator.com/item?id=41864632

A Cloudflare employee has said he'd like to hear from the maintainers of RSS readers if you Ctrl + F search @cloudflare.com which I think is the first line I've heard to someone at Cloudflare who cares about the issue.

A couple feeds I follow that haven't worked reliably in years are:

• https://fitbod.me/blog/category/product/feed/
• https://www.secretflying.com/posts/category/cities-countries/canada/toronto/feed/

I'm not sure if you have aggregate stats on 403s, but I expect a lot of them are going to be cloudflare related.

These will all load in a browser after answering a "Verify you are human" but they'll never load in CommaFeed because of the Cloudflare firewall.

I've contacted the site maintainers, but they're often not technically of allowing a subset of URLs through Cloudflare filtering.

Just thought I'd drop this note because it's the first time I've seen any hope of Cloudflare doing anything to improve this situation.