Cloudflare

[Carter-LL]

Keeps getting blocked by Cloudflare protection. Is this a common issue, and is there a fix?. Or am I doing something wrong? Thank you very much.

[bytedream]

Which login method are you using? (credentials, etp-rt cookie)

[Carter-LL]

I've tried all and same issue.

[Carter-LL]

[bytedream]

We had this issue already in the past but were able to bypass it. I also cannot reproduce it atm on my side :/
But I'm (now) aware that this issue reappears.

[bytedream]

You can try again later or tomorrow and let me know if the issue still exists. It might only be a short living cloudflare thing.

[Carter-LL]

Ah okay got it!

[Carter-LL]

Still having the same issue. Is there a way to use a proxy only for the application? Assuming it would work. Not very familiar with Cloudflare.

[Carter-LL]

You can try again later or tomorrow and let me know if the issue still exists. It might only be a short living cloudflare thing.

I don't know if you knew this but originally I used the exe from the current branch and compiled it myself which gave me cloudflare errors. But I just downloaded the one that was prebuilt which seems to be in go language and it doesn't throw any errors.

[bytedream]

The go binary uses other endpoints which (currently) aren't cloudflare protected. But this endpoints are deprecated and doesn't have the functionality of the ones used at v3.

[Askadelion]

In the V3 it also happens more often that the Cloudflare is triggered and you are then locked good 24h. In dev.06 I had so far no more but still happens. Reminds me of the problem of the GO version where the api has logged in too often instead of keeping the login.

[Malifico]

I also keep running into it, for like 3-4 days now

[KinDeDu]

It's been a long time that with crunchy-cli I can't log in to crunchyroll because of cloudfare, before this problem (I'm talking about a month and a half ago) I could use it anywhere besides my pc (google cloud, aws and other vps) but now it seems I can't bypass it without using proxy ( using only those like mullvad). It would be case to find a way to not trigger cloudfare every time. I hope you can take a moment to see what the problem is, @bytedream.

P.S I have tried using other similar programs on github out of curiosity and they seem not to trigger cloudfare but unfortunately they don't have all the fantastic functionality that crunchy-cli has.

[b1on1cdog]

I have same issue, its been a week and I can't use the binary due to cloudflare, would be good if I could at least solve the captcha challenge in a browser manually or connect it with an anti-captcha api

[bytedream]

For all users experiencing this issue, could you please provide the country you're doing the request from and make sure that you at least use the --credentials auth method?

---

From my tests with mullvad VPN I've experienced the following behavior for different countries:

• Sometimes bot protection: Australia, Canada, Finland, New Zealand
• Always bot protection: Moldova, Norway
• No bot protection: All other mullvad available countries

But note that this is not that representative as Cloudflare tends to block/check requests from VPN providers more often than from private homes.

[Malifico]

For all users experiencing this issue, could you please provide the country you're doing the request from and make sure that you at least use the --credentials auth method?

From my tests with mullvad VPN I've experienced the following behavior for different countries:

• Sometimes bot protection: Australia, Canada, Finland, New Zealand
• Always bot protection: Moldova, Norway
• No bot protection: All other mullvad available countries

But note that this is not that representative as Cloudflare tends to block/check requests from VPN providers more often than from private homes.

Did it with —credentials
Been using adresses from:
NL, GER and DK.
And have been fully blocked from connecting with the script from NL and GER (no proxy used tho)
NL worked fine for a week or so before it started to have cloudflare, then did it every couple of days and now its just fully blocked.
GER worked fully until it dident, there was no warnings, been blocked for weeks now.
DK still working just fine (for now)

[anon79678]

Used --credentials with a connection from Finland. It has been blocked for the last week now.

[KinDeDu]

Using the --credentials or --anonymous command:
as I mentioned in my previous answer.
Using service like Google Colab (Singapore) or AWS EC2 instances (Singapore, India, Australia, Germany) I always get blocked by Cloudfare, instead using VPN or Proxy Socks5 of the various EU nations (Italy, France, Austria, Spain) I don't get blocked .
With my local internet connection (Italy) I have no problems even using VPN or proxy from the EU nations mentioned above, I get blocked if I try to connect to (Singapore, India, Germany, Australia, Canada, Filandia, Poland, Ireland). I may try to connect to other nations but so far these are the problems I have had.

[bytedream]

After a bit testing (with this script/program) it seems that the used custom tls implementation causes most problems. A bit ironic because it was added to prevent cloudflare errors. But it seems that cf changed something and this implementation isn't necessary anymore (?).
I'll test a bit more and change the behavior if everything looks ok.

[bytedream]

I've pushed a update which should fix some cloudflare issues. Some locations (via mullvad) are still running into blocks but I haven't found out how to address this correctly.
It would be nice if some of you which have this issue could test the builds from here (use the binary for your system from the artifacts section) and reply if it works or not.

[Malifico]

Ill run some test tomorrow thanks for the update

[KinDeDu]

I tried the new build, the nations Germany, Ireland and India are no longer blocked. The rest of the ones I tried don't get through.

[anon79678]

Finland still blocked.

[bytedream]

After digging a bit deeper I think the used tls library (rustls) is too modern for cloudflare. Too modern means it doesn't support legacy/all protocols which cloudflare checks when doing request. Sadly the library which does the request (reqwest) only supports rustls or the system native tls (openssl on linux, schannel on windows, security-framework on mac) as backend. I wished it would support openssl on all major systems as it seems that openssl is the only tls backend which is able to bypass cloudflare.
I could replace reqwest with something that supports openssl among all platforms, like isahc, but I've tried this once in the past, and again, Windows causes so f... much problems with building and stuff that I want to avoid it. I could also fork reqwest and manually add openssl support on all platforms but I don't know how complicated this would be.
However the last commit added a fallback option for linux: If rustls doesn't work try again with the system native (openssl) library. Would be nice if someone with blocked linux could test it, the built binary can be found here.

[anon79678]

Would be nice if someone with blocked linux could test it, the built binary can be found here.

Tested it on Debian 11. No luck. Tried both credentials and etp-rt.

[KinDeDu]

I tried the build and did some tests, it doesn't seem to cause any problems with Cloudfare. All the countries I had tested are no longer blocked. Tested only on Linux, the one that was giving me problems.

[bytedream]

I've forked the native-tls library (crunchy-labs/rust-not-so-native-tls) and made openssl usable on Windows and Mac as well (couldn't test it on mac, I do not own a mac or macbook). On Windows the blocking still occurs, but less frequently, under Linux I have had no problems at all, even with different VPN locations.

[Frooastside]

Finland does still not work for me with neither the actions build you mentioned or the latest crunchy-cli v3.0.0-dev.15 build.

:: ✔ Logging in
:: expected value at line 1 column 1 at 1:1 (https://www.crunchyroll.com/auth/v1/token): : <!DOCTYPE html>
<html lang="en-US">
<head>
    <title>Just a moment...</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=Edge">
    <meta name="robots" content="noindex,nofollow">
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <link href="/cdn-cgi/styles/challenges.css" rel="stylesheet">

.....

[Frooastside]

I somewhat resolved the issue by using wireproxy. I couldn't use a vpn because that would block my ssh connection from the outside which would make my server inaccessible (i know there are workarounds but to complicated), i used it with the following config:

WGConfig = /root/ch-zrh-wg-404.conf

[Socks5]
BindAddress = 127.0.0.1:25344

where /root/ch-zrh-wg-404.conf is the wireguard config file of a mullvad server. Run it by wireproxy -c <config file>. Make sure to check if the server is blocked beforehand on another device (i tested it on my local machine and then pushed the config file to the server). And then used

crunchy-cli --proxy socks5://127.0.0.1:25344 archive -a ja-JP -o "./{series_name}/{series_name} - JP/Season {season_number}/{title} S{season_number}E{episode_number}.mkv" -m auto <URL>

[MoAlakhres]

I have a Kaspersky Subscription which can allow me to create a VPN config for a router. Will this work the same way? Also, I already tried switching over to the wiregaurd setting on my VPN app in windows and still get the cloudfare error.

[Frooastside]

I have actually no idea. I dont know the Kaspersky stuff. My solution was basically how to use a VPN on a remote Linux server non-globally so It doesn't block my ssh connection to it. The vpn connection itself must still be to a vpn provider that isn't blocked. If you have mullvad i can say that the Swizz ch-zrh-wg-404 server is not blocked. For other vpn providers you have to test that out yourself or even choose a different provider

[Lucy-2077]

Maybe should add the option to work with FlareSolverr ( https://github.com/FlareSolverr/FlareSolverr ) it should be better

[hajimekun]

With fixed shared IP (VPS not dedicated) it works only under VPN but i got the CF message in other ways, i was trying to scrape a page and a custom http session made using clouscraper can bypass the protection. Here is an example code, not related to downloading

import cloudscraper

class CustomHTTPSession(cloudscraper.CloudScraper):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def scrape_page(self, url):
        self.headers.update({'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36'})

        try:
            response = self.get(url)
            response.raise_for_status()

            print(response.text)

            return response
        except Exception as e:
            print(f"Error scraping page: {str(e)}")
            return None


if __name__ == "__main__":
    session = CustomHTTPSession()


    url = 'https://www.crunchyroll.com/'
    response = session.scrape_page(url)

[kennedy]

Out of curiosity, if you were to provide a "real" user-agent, like from https://whatmyuseragent.com, would you get the same cloudflare bot wall?

[bytedream]

Out of curiosity, if you were to provide a "real" user-agent, like from https://whatmyuseragent.com, would you get the same cloudflare bot wall?

This depends on the user agent used and the country the request comes from. I've made a little tool to check exactly this a while ago (https://github.com/crunchy-labs/cloudflare-bot-protect-check).