Skip to content
This repository has been archived by the owner on Jul 4, 2024. It is now read-only.

Cloudflare #104

Open
Carter-LL opened this issue Jan 9, 2023 · 34 comments
Open

Cloudflare #104

Carter-LL opened this issue Jan 9, 2023 · 34 comments
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@Carter-LL
Copy link

Carter-LL commented Jan 9, 2023

Keeps getting blocked by Cloudflare protection. Is this a common issue, and is there a fix?. Or am I doing something wrong? Thank you very much.

@bytedream
Copy link
Member

bytedream commented Jan 9, 2023

Which login method are you using? (credentials, etp-rt cookie)

@Carter-LL
Copy link
Author

I've tried all and same issue.

@Carter-LL
Copy link
Author

image

@bytedream bytedream added the bug Something isn't working label Jan 9, 2023
@bytedream
Copy link
Member

We had this issue already in the past but were able to bypass it. I also cannot reproduce it atm on my side :/
But I'm (now) aware that this issue reappears.

@bytedream
Copy link
Member

You can try again later or tomorrow and let me know if the issue still exists. It might only be a short living cloudflare thing.

@Carter-LL
Copy link
Author

Ah okay got it!

@Carter-LL
Copy link
Author

Still having the same issue. Is there a way to use a proxy only for the application? Assuming it would work. Not very familiar with Cloudflare.

@Carter-LL
Copy link
Author

You can try again later or tomorrow and let me know if the issue still exists. It might only be a short living cloudflare thing.

I don't know if you knew this but originally I used the exe from the current branch and compiled it myself which gave me cloudflare errors. But I just downloaded the one that was prebuilt which seems to be in go language and it doesn't throw any errors.

@bytedream
Copy link
Member

The go binary uses other endpoints which (currently) aren't cloudflare protected. But this endpoints are deprecated and doesn't have the functionality of the ones used at v3.

@bytedream bytedream added the help wanted Extra attention is needed label Jan 14, 2023
@Askadelion
Copy link

In the V3 it also happens more often that the Cloudflare is triggered and you are then locked good 24h. In dev.06 I had so far no more but still happens. Reminds me of the problem of the GO version where the api has logged in too often instead of keeping the login.

@Malifico
Copy link

I also keep running into it, for like 3-4 days now

@KinDeDu
Copy link

KinDeDu commented Jul 3, 2023

It's been a long time that with crunchy-cli I can't log in to crunchyroll because of cloudfare, before this problem (I'm talking about a month and a half ago) I could use it anywhere besides my pc (google cloud, aws and other vps) but now it seems I can't bypass it without using proxy ( using only those like mullvad). It would be case to find a way to not trigger cloudfare every time. I hope you can take a moment to see what the problem is, @bytedream.

P.S I have tried using other similar programs on github out of curiosity and they seem not to trigger cloudfare but unfortunately they don't have all the fantastic functionality that crunchy-cli has.

@b1on1cdog
Copy link

I have same issue, its been a week and I can't use the binary due to cloudflare, would be good if I could at least solve the captcha challenge in a browser manually or connect it with an anti-captcha api

@bytedream
Copy link
Member

For all users experiencing this issue, could you please provide the country you're doing the request from and make sure that you at least use the --credentials auth method?


From my tests with mullvad VPN I've experienced the following behavior for different countries:

  • Sometimes bot protection: Australia, Canada, Finland, New Zealand
  • Always bot protection: Moldova, Norway
  • No bot protection: All other mullvad available countries

But note that this is not that representative as Cloudflare tends to block/check requests from VPN providers more often than from private homes.

@Malifico
Copy link

Malifico commented Jul 5, 2023

For all users experiencing this issue, could you please provide the country you're doing the request from and make sure that you at least use the --credentials auth method?

From my tests with mullvad VPN I've experienced the following behavior for different countries:

  • Sometimes bot protection: Australia, Canada, Finland, New Zealand
  • Always bot protection: Moldova, Norway
  • No bot protection: All other mullvad available countries

But note that this is not that representative as Cloudflare tends to block/check requests from VPN providers more often than from private homes.

Did it with —credentials
Been using adresses from:
NL, GER and DK.
And have been fully blocked from connecting with the script from NL and GER (no proxy used tho)
NL worked fine for a week or so before it started to have cloudflare, then did it every couple of days and now its just fully blocked.
GER worked fully until it dident, there was no warnings, been blocked for weeks now.
DK still working just fine (for now)

@anon79678
Copy link

Used --credentials with a connection from Finland. It has been blocked for the last week now.

@KinDeDu
Copy link

KinDeDu commented Jul 6, 2023

Using the --credentials or --anonymous command:
as I mentioned in my previous answer.
Using service like Google Colab (Singapore) or AWS EC2 instances (Singapore, India, Australia, Germany) I always get blocked by Cloudfare, instead using VPN or Proxy Socks5 of the various EU nations (Italy, France, Austria, Spain) I don't get blocked .
With my local internet connection (Italy) I have no problems even using VPN or proxy from the EU nations mentioned above, I get blocked if I try to connect to (Singapore, India, Germany, Australia, Canada, Filandia, Poland, Ireland). I may try to connect to other nations but so far these are the problems I have had.

@bytedream
Copy link
Member

bytedream commented Jul 7, 2023

After a bit testing (with this script/program) it seems that the used custom tls implementation causes most problems. A bit ironic because it was added to prevent cloudflare errors. But it seems that cf changed something and this implementation isn't necessary anymore (?).
I'll test a bit more and change the behavior if everything looks ok.

@bytedream
Copy link
Member

I've pushed a update which should fix some cloudflare issues. Some locations (via mullvad) are still running into blocks but I haven't found out how to address this correctly.
It would be nice if some of you which have this issue could test the builds from here (use the binary for your system from the artifacts section) and reply if it works or not.

@Malifico
Copy link

Malifico commented Jul 8, 2023

Ill run some test tomorrow thanks for the update

@KinDeDu
Copy link

KinDeDu commented Jul 8, 2023

I tried the new build, the nations Germany, Ireland and India are no longer blocked. The rest of the ones I tried don't get through.

@anon79678
Copy link

Finland still blocked.

@bytedream
Copy link
Member

bytedream commented Jul 17, 2023

After digging a bit deeper I think the used tls library (rustls) is too modern for cloudflare. Too modern means it doesn't support legacy/all protocols which cloudflare checks when doing request. Sadly the library which does the request (reqwest) only supports rustls or the system native tls (openssl on linux, schannel on windows, security-framework on mac) as backend. I wished it would support openssl on all major systems as it seems that openssl is the only tls backend which is able to bypass cloudflare.
I could replace reqwest with something that supports openssl among all platforms, like isahc, but I've tried this once in the past, and again, Windows causes so f... much problems with building and stuff that I want to avoid it. I could also fork reqwest and manually add openssl support on all platforms but I don't know how complicated this would be.
However the last commit added a fallback option for linux: If rustls doesn't work try again with the system native (openssl) library. Would be nice if someone with blocked linux could test it, the built binary can be found here.

@anon79678
Copy link

Would be nice if someone with blocked linux could test it, the built binary can be found here.

Tested it on Debian 11. No luck. Tried both credentials and etp-rt.

@KinDeDu
Copy link

KinDeDu commented Jul 18, 2023

I tried the build and did some tests, it doesn't seem to cause any problems with Cloudfare. All the countries I had tested are no longer blocked. Tested only on Linux, the one that was giving me problems.

@bytedream
Copy link
Member

I've forked the native-tls library (crunchy-labs/rust-not-so-native-tls) and made openssl usable on Windows and Mac as well (couldn't test it on mac, I do not own a mac or macbook). On Windows the blocking still occurs, but less frequently, under Linux I have had no problems at all, even with different VPN locations.

@Frooastside
Copy link
Contributor

Finland does still not work for me with neither the actions build you mentioned or the latest crunchy-cli v3.0.0-dev.15 build.

:: ✔ Logging in
:: expected value at line 1 column 1 at 1:1 (https://www.crunchyroll.com/auth/v1/token): : <!DOCTYPE html>
<html lang="en-US">
<head>
    <title>Just a moment...</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=Edge">
    <meta name="robots" content="noindex,nofollow">
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <link href="/cdn-cgi/styles/challenges.css" rel="stylesheet">

.....

@Frooastside
Copy link
Contributor

Frooastside commented Jul 23, 2023

I somewhat resolved the issue by using wireproxy. I couldn't use a vpn because that would block my ssh connection from the outside which would make my server inaccessible (i know there are workarounds but to complicated), i used it with the following config:

WGConfig = /root/ch-zrh-wg-404.conf

[Socks5]
BindAddress = 127.0.0.1:25344

where /root/ch-zrh-wg-404.conf is the wireguard config file of a mullvad server. Run it by wireproxy -c <config file>. Make sure to check if the server is blocked beforehand on another device (i tested it on my local machine and then pushed the config file to the server). And then used

crunchy-cli --proxy socks5://127.0.0.1:25344 archive -a ja-JP -o "./{series_name}/{series_name} - JP/Season {season_number}/{title} S{season_number}E{episode_number}.mkv" -m auto <URL>

@MoAlakhres
Copy link

I have a Kaspersky Subscription which can allow me to create a VPN config for a router. Will this work the same way? Also, I already tried switching over to the wiregaurd setting on my VPN app in windows and still get the cloudfare error.

@Frooastside
Copy link
Contributor

Frooastside commented Aug 14, 2023

I have actually no idea. I dont know the Kaspersky stuff. My solution was basically how to use a VPN on a remote Linux server non-globally so It doesn't block my ssh connection to it. The vpn connection itself must still be to a vpn provider that isn't blocked. If you have mullvad i can say that the Swizz ch-zrh-wg-404 server is not blocked. For other vpn providers you have to test that out yourself or even choose a different provider

@Lucy-2077
Copy link

Maybe should add the option to work with FlareSolverr ( https://github.com/FlareSolverr/FlareSolverr ) it should be better

@hajimekun
Copy link

hajimekun commented Oct 9, 2023

With fixed shared IP (VPS not dedicated) it works only under VPN but i got the CF message in other ways, i was trying to scrape a page and a custom http session made using clouscraper can bypass the protection. Here is an example code, not related to downloading

import cloudscraper

class CustomHTTPSession(cloudscraper.CloudScraper):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def scrape_page(self, url):
        self.headers.update({'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36'})

        try:
            response = self.get(url)
            response.raise_for_status()

            print(response.text)

            return response
        except Exception as e:
            print(f"Error scraping page: {str(e)}")
            return None


if __name__ == "__main__":
    session = CustomHTTPSession()


    url = 'https://www.crunchyroll.com/'
    response = session.scrape_page(url)

@kennedy
Copy link
Contributor

kennedy commented Nov 5, 2023

Out of curiosity, if you were to provide a "real" user-agent, like from https://whatmyuseragent.com, would you get the same cloudflare bot wall?

@bytedream
Copy link
Member

Out of curiosity, if you were to provide a "real" user-agent, like from https://whatmyuseragent.com, would you get the same cloudflare bot wall?

This depends on the user agent used and the country the request comes from. I've made a little tool to check exactly this a while ago (https://github.com/crunchy-labs/cloudflare-bot-protect-check).

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests