Help solving a 503 on one feed

[skiingwiz]

I've got one feed out of about 40 that I can't get into News. It could be an issue on my system, and not with News. I'd appreciate any pointers on how to further debug this issue.
The feed is https://news.ycombinator.com/rss. Whenever I try to subscribe to it, it fails and I see this in the logs:

https://news.ycombinator.com/rss read error : Server error: GET https://news.ycombinator.com/rss resulted in a 503 Service Temporarily Unavailable response:\n\n\n <link rel="stylesheet" type="text/css" href="/news.css">\n <link rel="shortcut icon" href="/favicon.ico" (truncated...)\n

The thing that confuses me is that I can curl the feed just fine form the same host. I'm using the same User-Agent and Accept headers as News (or I think I am). The following curl works:

curl -H 'User-Agent: NextCloud-News/1.0' -H 'Accept: application/rss+xml, application/rdf+xml;q=0.8, application/atom+xml;q=0.6, application/xml;q=0.4, text/xml;q=0.4, /;q=0.2' https://news.ycombinator.com/rss

Running feedio manually (on another system in the same network) also works.

I'm not sure where to go from here. Suggestions welcome 😄 (I'm very happy with my transition to News, btw. It's a great piece of software.)

[SMillerDev]

Even if you solve your issue with the 503, that feed doesn't provide item identifiers so it would fail to import into news.

[Grotax]

Yep news can't handle that feed.

➜  bin git:(master) ./feedio check https://news.ycombinator.com/rss

reading https://news.ycombinator.com/rss
----------------------------------------

+--------------------------------------------------+------------+-----------+---------------------------+---------+------------+-----------+
| URL                                              | Accessible | readSince | Last modified             | # items | unique IDs | Date Flow |
+--------------------------------------------------+------------+-----------+---------------------------+---------+------------+-----------+
| https://news.ycombinator.com/rss                 | NOT OK     | NOT OK    | 2021-03-11T10:03:13+00:00 | 30      | NOT OK     | NOT OK    |
+--------------------------------------------------+------------+-----------+---------------------------+---------+------------+-----------+

The thing that confuses me is that I can curl the feed just fine form the same host. I'm using the same User-Agent and Accept headers as News (or I think I am). The following curl works:

Probably they block your nextcloud instance or the whole ip-range that your host is in. This can happen if they have automatic limits and track how often systems request that resource.

[skiingwiz]

Yep news can't handle that feed.

Ah, bummer. The feedio check command is something I'll note for future investigation.

The thing that confuses me is that I can curl the feed just fine form the same host. I'm using the same User-Agent and Accept headers as News (or I think I am). The following curl works:

Probably they block your nextcloud instance or the whole ip-range that your host is in. This can happen if they have automatic limits and track how often systems request that resource.

But the curl works, from the same host. So the IP isn't blocked. There must be something different between the GET that News is doing and the one that I'm doing manually with curl that the site is using to block.

[Grotax]

Oh ok I didn't read that correctly, maybe the request from news looks different

[skiingwiz]

Ok, I found the 503 issue. This URL responds with a 503 if multiple requests are sent too quickly. The reason this is happening in News is that in FeedServiceV2, it's calling Exporer->discover() followed immediately by FeedFetcher->fetch() for the same URL.

I've locally modified FeedServiceV2 to place a sleep(1) between those two calls and now News is able to add the feed.

So my question is, what's an acceptable way to fix/prevent this? I'm happy to provide a pull request, if there is a solution that project maintainers would accept. Maybe a sleep is OK. Or perhaps, only sleep/retry if there is an exception with the FeedFetcher->fetch() call.

[SMillerDev]

No, those are just quirks of the feed and I don't think we should make importing slower for everyone because of it.

[skiingwiz]

That's fair, but are there any solutions you'd consider? If a sleep/retry in a failure case is still no good, perhaps a way to tell news to add a feed without going through the discover process? (i.e. I'm adding the direct feed URL, no need to try to figure it out.)

While I agree that it's a quirk of the feed, it's a feed I want to read so I'll find a solution that works for me. I was hoping to be able to contribute something back to News, but if not I'll let this issue drop.

[SMillerDev]

If you can find a way to determine when a discover run is needed and when it isn't I'd happily review a pull request.