Feature request: daily max on addic7ed as they block IPs

[ghost]

Request
It would be great if it was possible to set a daily max on how many subtitles are downloaded from Addic7ed pr 24 hours.
(Possibly from UTC 00 - UTC 00? I don't know, guessing..)

Addic7ed
Addic7ed has started blocking IPs. Their limits are 40 subtitles pr 24h, if you are VIP you get 80 pr 24h. Their rules say they allow no scripts, only singular manual downloads.

This also means you can't log into addic7ed.com from a machine on a blocked IP (...). You need to be on a different IP to even access their contact form and send them a message, asking to be unblocked..

Debugging
I first discovered this a month ago as I noticed my subtitles were degrading in quality, checked the history (last 500 subtitles) in my kitana sub-zero menu and saw no addic7ed entries.
With help from my vpn provider, we found out my static IP withe them was blocked. I then switched to another IP + adjusted my settings in sub-zero to run less, check for better for a shorter period etc.

A few days ago it again happened on the new IP, only 4 weeks later after the former IP was blocked. I have now moved to a VPN IP on a machine in the neighboring country, as there are no more in the country I live in w Plex port available to forward. Which is unfortunate as ping is higher.

This is now the 4th IP I run Plex / sub-zero on. All the former 3 IPs cannot access addic7ed anymore (all of them worked originally). Every other IP I have tried can access addic7ed successfully (work VPN, other local machine on different VPN IP, local cafe, friends house - etc).

A month ago I found a post somewhere, I think on Addic7ed forum but not sure - which mentioned they block IPs. Can't find it just now :(

My setup
I am a VIP user with Addic7ed (donate every month). Sent them a request to unblock the original IP, no response.

Yes! I have anti captha enabled. I am on Sub-Zero 2.6.5.3223.

So! Before I move through the IP pool in my neighbor country, it would be awesome to be able to limit the number of downloads from Addic7ed pr day..

Curl and logs
From a blocked IP:

$ curl -IL https://www.addic7ed.com
curl: (7) Failed to connect to www.addic7ed.com port 443: Operation timed out

From a non blocked IP:

$ curl -IL https://www.addic7ed.com
HTTP/2 200
server: nginx
date: Tue, 07 Apr 2020 19:04:43 GMT
content-type: text/html
x-powered-by: PHP/5.4.16
set-cookie: PHPSESSID=p81nup9hodpstdbk5rkqrk56p4; path=/
expires: Thu, 19 Nov 1981 08:52:00 GMT
cache-control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
pragma: no-cache

Sub-zero log entries look like this:

2020-04-07 21:35:53,721 (70000e9b2000) :  ERROR (logger:38) - subliminal_patch.core: Provider 'addic7ed' timed out
2020-04-07 21:36:15,753 (700009573000) :  ERROR (logger:38) - subliminal_patch.core: Provider 'addic7ed' timed out
2020-04-07 21:36:17,818 (70000b98e000) :  ERROR (logger:38) - subliminal_patch.core: Provider 'addic7ed' timed out
2020-04-07 21:37:07,772 (700009573000) :  ERROR (logger:38) - subliminal_patch.core: Provider 'addic7ed' timed out
2020-04-07 21:37:26,459 (70000cd9d000) :  ERROR (logger:38) - subliminal_patch.core: Provider 'addic7ed' timed out

Possibly related issue
The last few lines in #722 looks like it could be the same issue. Provided "subzero plugin logs" means sub-zero history in the kitana user interface and not the actual text log files. If not, it is likely something different.

[kevindd992002]

This is really interesting. Thanks for tagging my case.

[pannal]

Interesting. I'll think of something.

[kevindd992002]

So addic7ed doesn't automatically unblock IP's after a certain amount of time? If that's the case, then shame on them. I understand why they do the blocking in the first place but it should only be temporarily.

[ghost]

Nope. The first IP was blocked months ago.

My guess is many have this issue but just didn't notice subtitles no longer come from addic7ed. You need to check the text log files to find the addic7ed timeouts. Even then, most will likely assume it's a temporary issue, like opensubtitles or subscene have now and then.

If you try log into addic7ed or curl their site, you will also find out of course. But unless you're a contributor or debugging this issue, why would you.

[ghost]

For debugging contest I must admit my usage seems to be far over addic7ed limits. I have logged in regularly recently and even with more limited settings in sub-zero addic7ed sometimes showed "380 of 80" subtitles downloaded in a day.
I have now reduced number of languages in sub-zero from 4 to to 2, right now it shows "142 of 80" downloaded today. Guessing I'll likely get blocked again in a week or two.

Part of the problem could be that Plex don't have options to choose which metadata to refresh. Once in a while I do "Refresh metadata" on everything from the last ~6 months, to get updated critic ratings. I don't know. I highly doubt Plex will change that though, their focus seems to be on music, dvr and linux platforms currently.

An option to hard limit addic7ed pr day would be amazing :)

[kevindd992002]

I only use English from the very beginning in SZ and I just a few tv shows (around 25 or so) in Plex. I'm not a heavy user in any way but there are a couple of times when I add a new tv show and have all episodes of it downloaded in one go.

[pannal]

Please try latest develop-2.6. It should add a daily limit (default 40, vip: 80 if enabled).
I'm currently resetting the limit at midnight, Germany/Berlin timezone as I believe they're German. Might be two hours off if they use UTC. If you can confirm that, that'd be great.

Edit: Hmm, might be Romania as well, judging by their WHOIS record. That'd be GMT+3.

[pannal]

Can you guys keep an eye on the profile page of yours on addic7ed as to when they reset the counters? Then please post the current UTC time and your timezone.

[kevindd992002]

Not sure if this same issue is the one causing my addic7ed issue as I explained here.

[ghost]

@kevindd992002 To verify:
Try do: curl -IL addic7ed.com in a terminal on the machine which has sub-zero installed.
Or: Open a browser and go to https://www.addic7ed.com/.
It has to be on the same machine which has sub-zero installed.

If you get a reply / can load their page, your IP is not blocked. If it times out, then there is a good chance your IP is blocked.

[ghost]

@pannal I have disabled addic7ed yesterday as my daily count was well over the 80 max. Been following my stats on their panel, looks like it is "trickling down" over time. 234, then 191, 189, 185 etc..

So. My current guess would be they calculate by "last 24h". I will let you know tomorrow if it looks more likely my stats rolled over to 0 at "24h since disabled" or "some EU timezone midnight".

[ghost]

Installed develop-2.6, getting this error:

2020-04-12 21:46:52,994 (700011c91000) :  ERROR (logger:34) - subliminal_patch.core: Unexpected error in provider 'addic7ed', Traceba
ck: Traceback (most recent call last):
  File "/Users/lumiere/Library/Application Support/Plex Media Server/Plug-ins/Sub-Zero.bundle/Contents/Libraries/Shared/subliminal_patch/core.py", line 257, in download_subtitle
    self[subtitle.provider_name].download_subtitle(subtitle)
  File "/Users/lumiere/Library/Application Support/Plex Media Server/Plug-ins/Sub-Zero.bundle/Contents/Libraries/Shared/subliminal_patch/providers/addic7ed.py", line 417, in download_subtitle
    if last_dl <= midnight:
TypeError: can't compare datetime.datetime to NoValue

[pannal]

Yikes, please try the latest commit!

looks like it is "trickling down" over time. 234, then 191, 189, 185 etc..

That's funky. Might they be really logging the downloads and let them expire 24h after any specific download? So no global "reset time"?

[kevindd992002]

@tvebax Here's the curl response:

HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Sun, 12 Apr 2020 22:24:35 GMT
Content-Type: text/html
Content-Length: 178
Connection: keep-alive
Location: https://addic7ed.com/

HTTP/2 301
server: nginx
date: Sun, 12 Apr 2020 22:24:36 GMT
content-type: text/html
content-length: 178
location: http://www.addic7ed.com/

HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Sun, 12 Apr 2020 22:24:37 GMT
Content-Type: text/html
Content-Length: 178
Connection: keep-alive
Location: https://www.addic7ed.com/

HTTP/2 200
server: nginx
date: Sun, 12 Apr 2020 22:24:38 GMT
content-type: text/html
x-powered-by: PHP/5.4.16
set-cookie: PHPSESSID=pojhddk8b2hjtqcm5cfmbmg7i0; path=/
expires: Thu, 19 Nov 1981 08:52:00 GMT
cache-control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
pragma: no-cache

So like I suspected, my IP is not blocked and I'm nowhere reaching the daily limit.

@pannal Yes, there is no global reset time. Here's an answer from their FAQ page:

[pannal]

OK, then I'll have to adjust my approach. Thanks for digging up that info!

[kevindd992002]

Yeah, so the limit expires per subtitle. I would guess an API query for the "current usage" of the user would help in optimizing the code?

@tvebax Also, based on that answer the IP throttle/block does expire depending on when the subtitles were downloaded. I'm wondering why your past VIP IP's were never unblocked along the way.

[pannal]

Hmm no, I'll take the easy way out and store the datetime in the cache, once a download happened. Then, before a download happens, I clear everything from the cache that's older than 24 hours.

No additional queries to addic7ed needed.

[kevindd992002]

Ok, that makes sense.

[pannal]

@tvebax last commit should now be a valid solution.

[kevindd992002]

So if in the cache there's still more than 40 subtitles with a datetime that's not older than 24 hours, will it skip addic7ed from being considered?

[pannal]

Yeah it'll pause for 4 hours after hitting the limit.

[kevindd992002]

The whole of SZ will pause or just the query to addic7ed? I'm assuming it's the latter.

[pannal]

I'll take this as a late April's fools joke.

[kevindd992002]

I'll take this as a late April's fools joke.

You bet. Sorry for the dumb question :)

[ghost]

@pannal It works! Using develop-2.6 branch. Addic7ed panel shows my user downloads pr day as 80 of 80 and Kitana SZ interface shows addic7ed as throttled. Perfect!

This is likely as it should be, but just in case it is not - FYI:
Sub-zero throws an error when reaching the limit. It looks like this is intentional and perhaps how triggering provider throttle works.

2020-04-13 20:13:43,369 (700005caf000) :  ERROR (logger:34) - subliminal_patch.core: Unexpected error in provider 'addic7ed', Traceback: Traceback (most recent call last):
  File "/Users/lumiere/Library/Application Support/Plex Media Server/Plug-ins/Sub-Zero.bundle/Contents/Libraries/Shared/subliminal_patch/core.py", line 257, in download_subtitle
    self[subtitle.provider_name].download_subtitle(subtitle)
  File "/Users/lumiere/Library/Application Support/Plex Media Server/Plug-ins/Sub-Zero.bundle/Contents/Libraries/Shared/subliminal_patch/providers/addic7ed.py", line 421, in download_subtitle
    raise DownloadLimitPerDayExceeded
DownloadLimitPerDayExceeded

Thank you for adding this feature. Really appreciate your attention, time and kindness.

[pannal]

Oh that's actually an oversight on my part. The exception itself shouldn't be shown, but it does no harm.

And yes, that's how the throttling system works - a provider throws an exception and the throttle mechanism decides what to do with that information. In this case it disables addic7ed for 4 hours.

My pleasure!

[kevindd992002]

@tvebax If it comes to a point that your SZ gets an exception from addic7ed (which means you tried querying even though you exceeded your limit of 80 already), wouldn't your IP get blocked like you mentioned earlier?

[pannal]

No, it does that when it hits the limit, before it tries another download:

Sub-Zero.bundle/Contents/Libraries/Shared/subliminal_patch/providers/addic7ed.py

Line 419 in 289f174

if amount >= cap:

I'll modify this so that this happens right after the download that hits the cap, so Addic7ed doesn't get hit with a search query before the limit is hit.

[kevindd992002]

I see. And the code takes consideration if the cap is either 40 (regular) or 80 (VIP), correct?

[pannal]

Of course.

[kevindd992002]

Ok. Where does that exception come from? Is it an internal SZ exception or from addic7ed themselves? I ask because addic7ed does allow the user to download more than the limit (like what's stated in @tvebax's reply here) and of course there wouldn't be an exception at that point.

EDIT: So it looks like it just takes a few minutes before the limit takes effect. And it makes sense for them to block an IP that downloads way over the limit before their system recognizes that the IP should be throttled.

[ghost]

Thanks for fixing this! Closing as the feature request is complete.