Subdivx not working

[santiagomartinez]

Hi all!
I just saw this issue: #2702
But in my case even after downloading tag v1.4.6-beta.2 I'm seeing in the logs:

2024-10-12 01:49:51|DEBUG   |subliminal_patch.providers.subdivx|Titles to look at: ['Requiem for a Dream', '夢之安魂曲', '梦的挽歌', '레퀴엠', 'Реквієм по мрії']|
2024-10-12 01:49:51|DEBUG   |urllib3.connectionpool          |https://www.subdivx.com:443 "GET /inc/gt.php?gt=1 HTTP/1.1" 200 None|
2024-10-12 01:49:51|DEBUG   |subliminal_patch.providers.subdivx|Query: Requiem for a Dream|
2024-10-12 01:49:51|DEBUG   |urllib3.connectionpool          |https://www.subdivx.com:443 "POST /inc/ajax.php HTTP/1.1" 200 None|
2024-10-12 01:49:51|DEBUG   |subliminal_patch.providers.subdivx|JSONDecodeError: b''|
2024-10-12 01:49:51|DEBUG   |urllib3.connectionpool          |https://www.subdivx.com:443 "GET /inc/gt.php?gt=1 HTTP/1.1" 200 None|
2024-10-12 01:49:51|DEBUG   |subliminal_patch.providers.subdivx|Query: Requiem for a Dream (2000)|
2024-10-12 01:49:51|DEBUG   |urllib3.connectionpool          |https://www.subdivx.com:443 "POST /inc/ajax.php HTTP/1.1" 200 None|
2024-10-12 01:49:51|DEBUG   |subliminal_patch.providers.subdivx|JSONDecodeError: b''|

Can anyone please confirm it is still working and it is on my side or it is not working again?

Thank you very much!

[morpheus65535]

Could it be a temporary bug (returning an empty response) since it seems to work for other users now? Let me know.

[santiagomartinez]

It seems to be down right now. I'll let you know with new tests results after it returns. Thanks!

[santiagomartinez]

Disregard the following, Bazarr is working now. Probably I was screwing something in the curls. Thanks!

---

It seems to be returning empty. I tested it with curl. I don't understand why because I compare it to the request payload seen in the browser where it works ok and both payloads match. I'm probably missing something

Token:
curl -X GET "https://subdivx.com/inc/gt.php?gt=1" -H "Content-Type: application/json"

Search:

curl -X POST "https://subdivx.com/inc/ajax.php" \
-H "Content-Type: application/json" \
-d '{"tabla": "resultados", "filtros": "", "buscar395a": "inside out", "token": "<TOKEN_VALUE>"}'

[morpheus65535]

Could it be that they implemented Cloudflare protection?

[alvarochvz]

Apparently?

[morpheus65535]

I can confirm they've implemented Cloudflare protection. I won't fight against them. If someone's interested, I'll accept a PR.

[alvarochvz]

Is this worth looking into?
https://trash-guides.info/Prowlarr/prowlarr-setup-flaresolverr/

I have never used it honestly i am not sure how well it will playout with bazarr 🤔

Is it even worth looking into?

[morpheus65535]

I've already tried to implement it without success but it may worth trying again.

[pombocaesar]

None of the other providers for Spanish are working correctly, it would be greatly appreciated if you could fix this one. Cheers

[LanusFac]

I’m having the same problem here. Bazarr just stopped working out of the blue when using SubDivx.

Has anyone managed to find the answer?

[morpheus65535]

I’m having the same problem here. Bazarr just stopped working out of the blue when using SubDivx.

Has anyone managed to find the answer?

There's no fix.

[pombocaesar]

I’m having the same problem here. Bazarr just stopped working out of the blue when using SubDivx.
Has anyone managed to find the answer?

There's no fix.

A couple weeks ago you mentioned trying without luck and it would probably worth another shot (I assume after this issue gaining popularity).
Does your last message mean you already retried and couldn't or that you are certain through some new information that it definitely can't be achieved?

[anderson-oki]

@pombocaesar

Putting in simple words, the provider implemented protection against non-humans to prevent automation. You can't simply bypass that. And we respect that they are protecting against robots.

[morpheus65535]

I agree with @anderson-oki. To better answer your question, I haven't retried and I din't have time to implement such feature for now.

[anderson-oki]

Just in case someone wants to invest time, i did a very quick check, and it seems that subdivx is not using the subliminal_patch http session that uses cloudscraper and it is simply using the requests.

One could try the luck with cloudscraper.

[morpheus65535]

Just in case someone wants to invest time, i did a very quick check, and it seems that subdivx is not using the subliminal_patch http session that uses cloudscraper and it is simply using the requests.

One could try the luck with cloudscraper.

I tried with no luck.

[dvillaseca]

Hi, after doing some testing, they are using the most basic detection with cloudflare on the get token request, if you use the cloudscraper library it can by bypassed
but it's tricky, only the get token and ajax request should be done with cloudscraper and not the rest
besides the new throttling they added

[morpheus65535]

@dvillaseca yeah won't be that easy. Even if I use RetryingCFSession() instead of Session(), I end up with a session token but /inc/ajax.php return a 403. Probably something from the cookiejar I must bring with me. I'll have to dig a little deeper but I'll take your input if you have some. :-)

[dvillaseca]

@dvillaseca yeah won't be that easy. Even if I use RetryingCFSession() instead of Session(), I end up with a session token but /inc/ajax.php return a 403. Probably something from the cookiejar I must bring with me. I'll have to dig a little deeper but I'll take your input if you have some. :-)

I managed to fix it and make it work but i'm a bit scare to share it, looks like they went to big depths in order to block automation, we maybe have to respect their effort and wait to see if they release a vip subscription system for automation like opensubtitles

[morpheus65535]

I managed to fix it and make it work but i'm a bit scare to share it, looks like they went to big depths in order to block automation, we maybe have to respect their effort and wait to see if they release a vip subscription system for automation like opensubtitles

It's up to you. I wont fight against them forever, I'll fix this once and if they continue to implement new security, I'll drop the towel and respect their choice.

[Airle]

This is a message shared by the Subdivx developer:

"Due to abuses caused by automated software, which overload the server, I had to implement some strategies to mitigate its impact."

There is no mention of any kind of VIP subscription.

[morpheus65535]

@Airle where did you see that message? Any way to contact site admins?

[Airle]

@Airle where did you see that message? Any way to contact site admins?

He left that message on his subdivx profile. His username is "braytac," so I guess you can contact him directly there. Look for him under "Miembros"

It also seems he has an account here: https://github.com/braytac.

[dvillaseca]

nice, let's contact him and ask for a vip feature looks like the official contact mail is info@subdivx.com

…

On Mon, Nov 18, 2024, 22:36 Airle ***@***.***> wrote: @Airle <https://github.com/Airle> where did you see that message? Any way to contact site admins? He left that message on his subdivx profile. His username is "braytac," so I guess you can contact him directly there. Look for him under "Miembros" It also seems he has an account here: https://github.com/braytac. — Reply to this email directly, view it on GitHub <#2714 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABFPSSLLQA3MNSBTCXIY6ST2BKIZHAVCNFSM6AAAAABPZ5ONKWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBUGUZDIMRVGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[morpheus65535]

I don't have an account on subdivx so it will be hard to achieve on my side.

[pombocaesar]

I do and I'm also Argentinian (he is, based on .ar in his account TLD) so I'll reach out to him now.

…

On Tue, 19 Nov 2024, 5:25 pm morpheus65535, ***@***.***> wrote: I don't have an account on subdivx so it will be hard to achieve on my side. — Reply to this email directly, view it on GitHub <#2714 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH3PFEFLU7Q6ENVJYFXWYED2BK4RZAVCNFSM6AAAAABPZ5ONKWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBUGY3TSOJYG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[morpheus65535]

I do and I'm also Argentinian (he is, based on .ar in his account TLD) so I'll reach out to him now.

Thanks, tell him we don't want to fight and we understand and respect his choice. We'll be happy to discuss with him on Discord.

[alvarochvz]

Hey so if there is a possibility of this being a premium feature let us know -- i would also be on board if you need hands to code such feature.

[braytac]

I can confirm they've implemented Cloudflare protection. I won't fight against them. If someone's interested, I'll accept a PR.

Greetings!
I am a developer of subdivx (not the owner).
One of the things I had proposed 2 years ago was to make an API to work with subliminal, but I still didn't have time.
I had even less time in the last few months, because half the time I had to build different strategies to mitigate an extremely abusive consumption of the server. I had the mysqld process at 100% all the time. I had load average at 30!!
First I started to filter for requests that came from the "subliminal" agent (exactly the one I wanted to run, since it is the one I use the most). But after a while the consumption came back and was much worse.
No, we don't have any protection from cloudflare. I simply implemented a penalty strategy for each query identified as abusive. The more retries there are in a period of time defined as short, the waiting time is multiplied x2, without returning results.
That way, I was able to get the server back to a normal load average of ~1.5, where everything works fine for users who download manually from the web.
I have no problem coordinating with you any special request dedicated to automated software, which is less expensive for the DB engine, and so that there are no successive requests in a very short period of time.
For movies, there shouldn't be much of a problem. For series, you could also adapt to iterate on a single response in json, for a given season. Having the IDs of each subtitle already, you wouldn't need to query the database again for each specific chapter.

[dvillaseca]

@morpheus65535 hello, have you coordinated something like this in the past?
I'm not completely familiarized on how bazarr queries each provider when searching for tv show subtitles,
I have seen that bazarr makes tons for search queries, there is a way of reducing the amount of queries made so we don't overload the server?

[morpheus65535]

@braytac sorry for the delay responding to your message. Of course we can work together to reduce the footprint of Bazarr on your website. Here's some suggestions:

• We stop using random user-agent and provide something that you can catch and work with for throttling or blocking.
• We make sure to impose a delay between queries (5 seconds?) to limit number of queries sent by a single user.
• If you implement x-rate-limit headers, we can follow what they instruct.
• Without going down the full API path, if you can provide 2 GET endpoints that we could call using imdb number and, for series, season and episode numbers, we could use this and simply get json back in response. This would prevent a lot of js, css, jpeg and other stuff being downloaded which is totally useless.
• We could cache answer for non-existant movies or series and prevent multiple queries for the same stuff from reaching your server within 24 hours.

Anything that make sense to you here? If you want to discuss further, easiest way may be on our Discord dev-community channel.

Thanks!