Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cloudflare CAPTCHA error with outdated python dependencies #460

Closed
GiovanH opened this issue Oct 31, 2019 · 5 comments
Closed

Cloudflare CAPTCHA error with outdated python dependencies #460

GiovanH opened this issue Oct 31, 2019 · 5 comments

Comments

@GiovanH
Copy link
Contributor

GiovanH commented Oct 31, 2019

I get a cloudflare captcha error when I run the python version of the application, but not when I run the bundled EXE version.

Issue reproduced using Patreon:

EXE:

$ ./gallery-dl-dev.exe https://www.patreon.com/giovan --cookies cookies.txt -v
[gallery-dl][debug] Version 1.11.0-dev
[gallery-dl][debug] Python 3.7.3 - Windows-10-10.0.17763
[gallery-dl][debug] requests 2.22.0 - urllib3 1.25.6
[gallery-dl][debug] Starting DownloadJob for 'https://www.patreon.com/giovan'
[gallery-dl][debug] Updating urllib3 ciphers
[patreon][debug] Using PatreonCreatorExtractor for 'https://www.patreon.com/giovan'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.patreon.com:443
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /giovan HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /api/posts?include=user,images,attachments,user_defined_tags,campaign,poll.choices,poll.current_user_responses.user,poll.current_user_responses.choice,poll.current_user_responses.poll,access_rules.tier.null&fields%5Bpost%5D=change_visibility_at,comment_count,content,current_user_can_delete,current_user_can_view,current_user_has_liked,embed,image,is_paid,like_count,min_cents_pledged_to_view,post_file,published_at,patron_count,patreon_url,post_type,pledge_url,thumbnail_url,teaser_text,title,upgrade_url,url,was_posted_by_campaign_owner&fields%5Buser%5D=image_url,full_name,url&fields%5Bcampaign%5D=avatar_photo_url,earnings_visibility,is_nsfw,is_monthly,name,url&fields%5Baccess_rule%5D=access_rule_type,amount_cents&sort=-published_at&filter%5Bis_draft%5D=false&filter%5Bcontains_exclusive_posts%5D=true&filter%5Bcampaign_id%5D=1125369&json-api-use-default-includes=false&json-api-version=1.0 HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /api/user/4029845 HTTP/1.1" 200 None
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): c10.patreonusercontent.com:443
[urllib3.connectionpool][debug] https://c10.patreonusercontent.com:443 "GET /3/eyJhIjoxLCJwIjoxfQ%3D%3D/patreon-media/p/post/17932987/e833710702b54a37a28d20585a500934/1.png?token-time=1572566400&token-hash=pFB6T8fjgznoZ0y6iotdNXWzgnR2MuJ6LQSfPgbALzI%3D HTTP/1.1" 200 13704
* .\gallery-dl\patreon\Giovan\17932987_exclusive to patreons_01.png

Python:

$ python3 gallery_dl/__main__.py https://www.patreon.com/giovan --cookies cookies.txt -v
[gallery-dl][debug] Version 1.11.0-dev - Git HEAD: 8361d87
[gallery-dl][debug] Python 3.7.0b4 - Windows-10-10.0.17763-SP0
[gallery-dl][debug] requests 2.22.0 - urllib3 1.23
[gallery-dl][debug] Starting DownloadJob for 'https://www.patreon.com/giovan'
[gallery-dl][debug] Updating urllib3 ciphers
[patreon][debug] Using PatreonCreatorExtractor for 'https://www.patreon.com/giovan'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.patreon.com:443
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /giovan HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /api/posts?include=user,images,attachments,user_defined_tags,campaign,poll.choices,poll.current_user_responses.user,poll.current_user_responses.choice,poll.current_user_responses.poll,access_rules.tier.null&fields[post]=change_visibility_at,comment_count,content,current_user_can_delete,current_user_can_view,current_user_has_liked,embed,image,is_paid,like_count,min_cents_pledged_to_view,post_file,published_at,patron_count,patreon_url,post_type,pledge_url,thumbnail_url,teaser_text,title,upgrade_url,url,was_posted_by_campaign_owner&fields[user]=image_url,full_name,url&fields[campaign]=avatar_photo_url,earnings_visibility,is_nsfw,is_monthly,name,url&fields[access_rule]=access_rule_type,amount_cents&sort=-published_at&filter[is_draft]=false&filter[contains_exclusive_posts]=true&filter[campaign_id]=1125369&json-api-use-default-includes=false&json-api-version=1.0 HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.patreon.com:443 "GET /api/user/4029845 HTTP/1.1" 403 None
[patreon][warning] Cloudflare CAPTCHA
[patreon][error] HttpError: '403 Forbidden' for 'https://www.patreon.com/api/user/4029845'

My requirements are all up-to-date, too:

$ python3 -m pip install --upgrade -r requirements.txt
Requirement already up-to-date: requests>=2.11.0 in c:\program files\python37\lib\site-packages (from -r requirements.txt (line 1)) (2.22.0)
Requirement already satisfied, skipping upgrade: idna<2.9,>=2.5 in c:\program files\python37\lib\site-packages (from requests>=2.11.0->-r requirements.txt (line 1)) (2.7)
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in c:\program files\python37\lib\site-packages (from requests>=2.11.0->-r requirements.txt (line 1)) (2018.4.16)
Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\program files\python37\lib\site-packages (from requests>=2.11.0->-r requirements.txt (line 1)) (1.23)
Requirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in c:\program files\python37\lib\site-packages (from requests>=2.11.0->-r requirements.txt (line 1)) (3.0.4)
@mikf
Copy link
Owner

mikf commented Oct 31, 2019

My requirements are all up-to-date, too:

According to pip, but its dependency management is really bad. The urllib3 package in your Python environment is rather old (1.23 vs 1.25.6), and so is probably pyOpenSSL. Try updating those two and see if that helps:

$ python3 -m pip install --upgrade urllib3 pyOpenSSL

If you're still getting a Cloudflare CAPTCHA afterwards, you might have to upgrade their dependencies as well.

@GiovanH
Copy link
Contributor Author

GiovanH commented Oct 31, 2019 •
edited
Loading

I was able to upgrade urllib3 from 1.23 to 1.25.6, but that doesn't change the program output.

$ python3 -m pip install --upgrade urllib3 pyOpenSSL
Collecting urllib3
  Downloading https://files.pythonhosted.org/packages/e0/da/55f51ea951e1b7c63a579c09dd7db825bb730ec1fe9c0180fc77bfb31448/urllib3-1.25.6-py2.py3-none-any.whl (125kB)
     |████████████████████████████████| 133kB 6.8MB/s
Requirement already up-to-date: pyOpenSSL in c:\program files\python37\lib\site-packages (19.0.0)
Requirement already satisfied, skipping upgrade: six>=1.5.2 in c:\program files\python37\lib\site-packages (from pyOpenSSL) (1.11.0)
Requirement already satisfied, skipping upgrade: cryptography>=2.3 in c:\users\seth\appdata\roaming\python\python37\site-packages (from pyOpenSSL) (2.4.2)
Requirement already satisfied, skipping upgrade: idna>=2.1 in c:\program files\python37\lib\site-packages (from cryptography>=2.3->pyOpenSSL) (2.7)
Requirement already satisfied, skipping upgrade: asn1crypto>=0.21.0 in c:\users\seth\appdata\roaming\python\python37\site-packages (from cryptography>=2.3->pyOpenSSL) (0.24.0)
Requirement already satisfied, skipping upgrade: cffi!=1.11.3,>=1.7 in c:\program files\python37\lib\site-packages (from cryptography>=2.3->pyOpenSSL) (1.11.5)
Requirement already satisfied, skipping upgrade: pycparser in c:\program files\python37\lib\site-packages (from cffi!=1.11.3,>=1.7->cryptography>=2.3->pyOpenSSL) (2.18)
Installing collected packages: urllib3
  Found existing installation: urllib3 1.23
    Uninstalling urllib3-1.23:
      Successfully uninstalled urllib3-1.23
Successfully installed urllib3-1.25.6

Upgrading the dependencies manually, though, did solve the problem:

$ python3 -m pip install --upgrade cryptography six asn1crypto cffi idna pycparser halo
Collecting cryptography
  Using cached https://files.pythonhosted.org/packages/a4/9c/9ad11ad293bb75d37fa63a0d1f47ae7f6ccda32bc7452bf2ffb788ca753f/cryptography-2.8-cp37-cp37m-win_amd64.whl
Requirement already up-to-date: six in c:\program files\python37\lib\site-packages (1.12.0)
Collecting asn1crypto
  Using cached https://files.pythonhosted.org/packages/6b/8c/ff300484eca90b397d919408619fae479965bdd8a1df3d6d08d58e491da5/asn1crypto-1.2.0-py2.py3-none-any.whl
Requirement already up-to-date: cffi in c:\program files\python37\lib\site-packages (1.13.1)
Collecting idna
  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Requirement already up-to-date: pycparser in c:\program files\python37\lib\site-packages (2.19)
Collecting halo
  Downloading https://files.pythonhosted.org/packages/3f/40/a9e9d61a5e7d1946acbf0b6d6e35e87dfd0cc443cdb90e9f3f0066d0d0ef/halo-0.0.28-py3-none-any.whl
Requirement already satisfied, skipping upgrade: spinners>=0.0.23 in c:\users\seth\appdata\roaming\python\python37\site-packages (from halo) (0.0.23)
Requirement already satisfied, skipping upgrade: termcolor>=1.1.0 in c:\program files\python37\lib\site-packages\termcolor-1.1.0-py3.7.egg (from halo) (1.1.0)
Requirement already satisfied, skipping upgrade: cursor>=1.2.0 in c:\users\seth\appdata\roaming\python\python37\site-packages (from halo) (1.2.0)
Collecting log-symbols>=0.0.13 (from halo)
  Downloading https://files.pythonhosted.org/packages/28/5d/d710c38be68b0fb54e645048fe359c3904cc3cb64b2de9d40e1712bf110c/log_symbols-0.0.14-py3-none-any.whl
Requirement already satisfied, skipping upgrade: colorama>=0.3.9 in c:\users\seth\appdata\roaming\python\python37\site-packages (from halo) (0.3.9)
Installing collected packages: cryptography, asn1crypto, idna, log-symbols, halo
  Found existing installation: cryptography 2.4.2
    Uninstalling cryptography-2.4.2:
      Successfully uninstalled cryptography-2.4.2
  Found existing installation: asn1crypto 0.24.0
    Uninstalling asn1crypto-0.24.0:
      Successfully uninstalled asn1crypto-0.24.0
  Found existing installation: idna 2.7
    Uninstalling idna-2.7:
      Successfully uninstalled idna-2.7
  Found existing installation: log-symbols 0.0.11
    Uninstalling log-symbols-0.0.11:
      Successfully uninstalled log-symbols-0.0.11
  Found existing installation: halo 0.0.18
    Uninstalling halo-0.0.18:
      Successfully uninstalled halo-0.0.18
Successfully installed asn1crypto-1.2.0 cryptography-2.8 halo-0.0.28 idna-2.8 log-symbols-0.0.14

(halo included because my installed version of halo wanted an outdated version of six; that shouldn't be relevant to this project)
A solution might be to add the required versions of the dependencies to requirements.txt

@GiovanH GiovanH changed the title Cloudflare CAPTCHA with python version Cloudflare CAPTCHA error with outdated python dependencies Oct 31, 2019
@mikf
Copy link
Owner

mikf commented Oct 31, 2019

The important thing here, I think, was updating cryptography from 2.4.2 to 2.8 to get up-to-date support for SSL/TLS ciphers, which is what's triggering the Cloudflare captcha by the way.

This package is a dependency of pyOpenSSL, which is an optional dependency of gallery-dl and is only sometimes required for Patreon and ArtStation. Adding this to requirements.txt would be overkill, but an extra installation target (pip install gallery-dl[extra]) might be in order.

@Hrxn
Copy link
Contributor

Hrxn commented Oct 31, 2019

I think you should be able to see all outdated packages (including dependencies) with
pip list --outdated

I do it this way from time to time and then --upgrade all listed packages, and I think I never had any outdated deps so far.

mikf added a commit that referenced this issue Nov 3, 2019
@mikf
update setup.py
e782b4b 
- add 'cloudflare' and 'video' extra install target (#460)
  (e.g. 'pip install gallery-dl[cloudflare]')
- remove py2exe and distutils code
- get version without calling exec()
- inline variables
@mikf
Copy link
Owner

mikf commented Nov 3, 2019

There are now 2 extra "requirement targets" for gallery-dl: cloudflare and video. Once the next version gets released and uploaded to PyPI, you can do gallery-dl[cloudflare] to install all packages required to bypass Cloudflare CAPTCHAS. (pyOpenSSL and cryptography)

@mikf mikf closed this as completed Nov 3, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mikf @GiovanH @Hrxn