SOCKS proxy support #9287

yan12125 · 2016-04-23T13:46:10Z

Implements #402.

I adapt the public domain version of socks.py proposed by @bluec0re at #305 (comment).

To run test_socks.py, you need to run two SOCKS server on two different public IPs. For example, create test/local_parameters.json with the following contents:

{
    "primary_proxy": "socks5://127.0.0.1/",
    "primary_server_ip": "server1_ip",
    "secondary_proxy": "socks5://127.0.0.1:1090/",
    "secondary_server_ip": "server2_ip"
}

And then run two SSH servers: ssh server1_ip -D 1080, ssh server2_ip -D 1090

(The step numbers are going wrong - blame Github Flavored Markdown :) )
And my test results:

$ bash pythons.sh test/test_socks.py -v
+ for interp in python2.6 python2.7 python3.2 python3.3 python3.4 python3.5 python3.6 pypy pypy3 jython
+ which python2.6
+ python2.6 -W error test/test_socks.py -v
test_proxy_http (__main__.TestSocks) ... ok
test_proxy_https (__main__.TestSocks) ... ok
test_secondary_proxy_http (__main__.TestSocks) ... ok
test_secondary_proxy_https (__main__.TestSocks) ... ok

----------------------------------------------------------------------
Ran 4 tests in 4.722s

OK
+ for interp in python2.6 python2.7 python3.2 python3.3 python3.4 python3.5 python3.6 pypy pypy3 jython
+ which python2.7
+ python2.7 -W error test/test_socks.py -v
test_proxy_http (__main__.TestSocks) ... ok
test_proxy_https (__main__.TestSocks) ... ok
test_secondary_proxy_http (__main__.TestSocks) ... ok
test_secondary_proxy_https (__main__.TestSocks) ... ok

----------------------------------------------------------------------
Ran 4 tests in 4.097s

OK
+ for interp in python2.6 python2.7 python3.2 python3.3 python3.4 python3.5 python3.6 pypy pypy3 jython
+ which python3.2
+ python3.2 -W error test/test_socks.py -v
test_proxy_http (__main__.TestSocks) ... ok
test_proxy_https (__main__.TestSocks) ... ok
test_secondary_proxy_http (__main__.TestSocks) ... ok
test_secondary_proxy_https (__main__.TestSocks) ... ok

----------------------------------------------------------------------
Ran 4 tests in 4.108s

OK
+ for interp in python2.6 python2.7 python3.2 python3.3 python3.4 python3.5 python3.6 pypy pypy3 jython
+ which python3.3
+ continue
+ for interp in python2.6 python2.7 python3.2 python3.3 python3.4 python3.5 python3.6 pypy pypy3 jython
+ which python3.4
+ continue
+ for interp in python2.6 python2.7 python3.2 python3.3 python3.4 python3.5 python3.6 pypy pypy3 jython
+ which python3.5
+ python3.5 -W error test/test_socks.py -v
test_proxy_http (__main__.TestSocks) ... ok
test_proxy_https (__main__.TestSocks) ... ok
test_secondary_proxy_http (__main__.TestSocks) ... ok
test_secondary_proxy_https (__main__.TestSocks) ... ok

----------------------------------------------------------------------
Ran 4 tests in 4.035s

OK
+ for interp in python2.6 python2.7 python3.2 python3.3 python3.4 python3.5 python3.6 pypy pypy3 jython
+ which python3.6
+ python3.6 -W error test/test_socks.py -v
test_proxy_http (__main__.TestSocks) ... ok
test_proxy_https (__main__.TestSocks) ... ok
test_secondary_proxy_http (__main__.TestSocks) ... ok
test_secondary_proxy_https (__main__.TestSocks) ... ok

----------------------------------------------------------------------
Ran 4 tests in 4.049s

OK
+ for interp in python2.6 python2.7 python3.2 python3.3 python3.4 python3.5 python3.6 pypy pypy3 jython
+ which pypy
+ pypy -W error test/test_socks.py -v
test_proxy_http (__main__.TestSocks) ... ok
test_proxy_https (__main__.TestSocks) ... ok
test_secondary_proxy_http (__main__.TestSocks) ... ok
test_secondary_proxy_https (__main__.TestSocks) ... ok

----------------------------------------------------------------------
Ran 4 tests in 4.137s

OK
+ for interp in python2.6 python2.7 python3.2 python3.3 python3.4 python3.5 python3.6 pypy pypy3 jython
+ which pypy3
+ pypy3 -W error test/test_socks.py -v
test_proxy_http (__main__.TestSocks) ... ok
test_proxy_https (__main__.TestSocks) ... ok
test_secondary_proxy_http (__main__.TestSocks) ... ok
test_secondary_proxy_https (__main__.TestSocks) ... ok

----------------------------------------------------------------------
Ran 4 tests in 4.131s

OK
+ for interp in python2.6 python2.7 python3.2 python3.3 python3.4 python3.5 python3.6 pypy pypy3 jython
+ which jython
+ jython -W error test/test_socks.py -v
test_proxy_http (__main__.TestSocks) ... ok
test_proxy_https (__main__.TestSocks) ... ERROR
test_secondary_proxy_http (__main__.TestSocks) ... ok
test_secondary_proxy_https (__main__.TestSocks) ... ERROR

======================================================================
ERROR: test_proxy_https (__main__.TestSocks)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_socks.py", line 43, in test_proxy_https
    self.assertEqual(
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/YoutubeDL.py", line 1942, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/opt/jython/Lib/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/opt/jython/Lib/urllib2.py", line 448, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/opt/jython/Lib/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/utils.py", line 912, in https_open
    return self.do_open(functools.partial(
  File "/opt/jython/Lib/urllib2.py", line 1197, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 1] No subject alternative DNS name matching yt-dl.org found. (javax.net.ssl.SSLHandshakeException: General SSLEngine problem)>

======================================================================
ERROR: test_secondary_proxy_https (__main__.TestSocks)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_socks.py", line 65, in test_secondary_proxy_https
    self.assertEqual(
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/YoutubeDL.py", line 1942, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/opt/jython/Lib/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/opt/jython/Lib/urllib2.py", line 448, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/opt/jython/Lib/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/utils.py", line 912, in https_open
    return self.do_open(functools.partial(
  File "/opt/jython/Lib/urllib2.py", line 1197, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 1] No subject alternative DNS name matching yt-dl.org found. (javax.net.ssl.SSLHandshakeException: General SSLEngine problem)>

----------------------------------------------------------------------
Ran 4 tests in 3.413s

FAILED (errors=2)

HTTPS over SOCKS fails on Jython. I don't know why. Direct connection to https://yt-dl.org/ip is OK in Jython. I'll open a new issue and mark it as "known bugs" if this PR get merged.

https://gist.github.com/bluec0re/cafd3764412967417fd3

yan12125 · 2016-04-23T14:12:34Z

Update: direct connection to yt-dl.org in Jython also fails:

$ jython
Jython 2.7.1b3 (default:87534ec6252a+, Apr 16 2016, 22:39:23) 
[OpenJDK 64-Bit Server VM (Oracle Corporation)] on java1.8.0_77
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib2
>>> urllib2.urlopen('https://yt-dl.org/')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/jython/Lib/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/opt/jython/Lib/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/opt/jython/Lib/urllib2.py", line 448, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/opt/jython/Lib/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/opt/jython/Lib/urllib2.py", line 1239, in https_open
    return self.do_open(httplib.HTTPSConnection, req,
  File "/opt/jython/Lib/urllib2.py", line 1197, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 1] No subject alternative DNS name matching yt-dl.org found. (javax.net.ssl.SSLHandshakeException: General SSLEngine problem)>
>>>

From Wireshark it's shown a *.aries.uberspace.de certificate is returned. Maybe a Jython bug.

OliverUv · 2016-04-30T13:33:26Z

Thanks a bunch for this. Since I'm in China, don't have much choice but to run this branch. Hope it gets integrated soon.

EDIT: specifying --proxy "socks5://127.0.0.1/" is not enough if the socks server listens on 1080. One must specify --proxy "socks5://127.0.0.1:1080/"

yan12125 · 2016-04-30T14:26:18Z

In my codes the default port for SOCKS proxy is 1080. Can you paste the full verbose result from the following command?

youtube-dl <whatever URL> --verbose --proxy "socks5://127.0.0.1/"

By the way, don't edit comments except minor changes (typo etc.). The buggy Github does not send notifications about edited comments.

OliverUv · 2016-04-30T14:33:52Z

Thanks, I'll keep in mind not to use the edit feature for adding anything important in the future. Here is the output. I get other errors when I specify port 1080, but at least it gets past this one.

> ~/Projects/youtube-dl/bin/youtube-dl 'https://www.youtube.com/watch?v=UeSb1BMLGNc'  --verbose --proxy "socks5://127.0.0.1/"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'https://www.youtube.com/watch?v=UeSb1BMLGNc', u'--verbose', u'--proxy', u'socks5://127.0.0.1/']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.04.24
[debug] Python version 2.7.6 - Linux-3.16.0-38-generic-x86_64-with-LinuxMint-17.2-rafaela
[debug] exe versions: avconv 9.18-6, avprobe 9.18-6, rtmpdump 2.4
[debug] Proxy map: {u'http': u'socks5://127.0.0.1/', u'https': u'socks5://127.0.0.1/'}
[youtube] UeSb1BMLGNc: Downloading webpage
ERROR: Unable to download webpage: <urlopen error [Errno 111] Connection refused> (caused by URLError(error(111, 'Connection refused'),))
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 388, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 1942, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/utils.py", line 859, in https_open
    req, **kwargs)
  File "/usr/lib/python2.7/urllib2.py", line 1184, in do_open
    raise URLError(err)

OliverUv · 2016-04-30T14:41:54Z

For reference, I get stuck here when I add the port, and can see in my socks server logs that youtube-dl does not attempt to use the socks proxy at all.

> ~/Projects/youtube-dl/bin/youtube-dl 'https://www.youtube.com/watch?v=UeSb1BMLGNc'  --verbose --proxy "socks5://127.0.0.1:1080/"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'https://www.youtube.com/watch?v=UeSb1BMLGNc', u'--verbose', u'--proxy', u'socks5://127.0.0.1:1080/']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.04.24
[debug] Python version 2.7.6 - Linux-3.16.0-38-generic-x86_64-with-LinuxMint-17.2-rafaela
[debug] exe versions: avconv 9.18-6, avprobe 9.18-6, rtmpdump 2.4
[debug] Proxy map: {u'http': u'socks5://127.0.0.1:1080/', u'https': u'socks5://127.0.0.1:1080/'}
[youtube] UeSb1BMLGNc: Downloading webpage

yan12125 · 2016-04-30T14:45:15Z

Can you run the following command:

strace -f ~/Projects/youtube-dl/bin/youtube-dl 'https://www.youtube.com/watch?v=UeSb1BMLGNc'  --verbose --proxy "socks5://127.0.0.1/" >& strace.txt

And upload strace.txt?

This file may contain sensitive data (your IP, username etc.). Check its contents before uploading.

dstftw · 2016-05-01T01:33:49Z

youtube_dl/compat.py

+        return struct.unpack(spec, *args)
+else:
+    struct_pack = struct.pack
+    struct_unpack = struct.unpack


As moving to compat.py why not prefix with conventional compat_?

Maybe shlex_quote and subprocess_check_output should be renamed, too?

Maybe but not as part of this PR. subprocess_check_output also seems to be unused at all.

OliverUv · 2016-05-01T04:18:55Z

Here is the strace you asked for. I've never used strace to debug, so if you have time (or links to good articles) I'd love to see what parts of it you'll be looking at for your debugging.

yan12125 · 2016-05-01T09:29:42Z

@OliverUv From your logs there are at least two youtube-dl installations. One is at /usr/local/lib/python2.7/dist-packages/youtube_dl/ and another is at ~/Projects/youtube-dl/bin/youtube-dl. Please remove the former to prevent mysterious problems.

OliverUv · 2016-05-01T17:52:31Z

Very sorry for the noise @yan12125 - your suggestion solved everything. I was unaware that python would reach out to the globally installed instance when I tried to run (the non-built) local instance. I've now also learned that I need to python setup.py build to properly run distutils depending projects. Thanks for the help, and thanks for great software!

yan12125 · 2016-05-05T09:11:43Z

Use srelay to test socks.py with local servers. Let's see the CI result.

dstftw · 2016-05-07T19:27:39Z

youtube_dl/socks.py

+                self.close()
+                raise Socks5Error(Socks5Error.ERR_GENERAL_FAILURE)
+        elif method == Socks5Auth.AUTH_NONE:
+            pass


I guess this elif clause can be safely omitted.

dstftw · 2016-05-07T19:42:39Z

I've not had a chance to take a deep look yet but I guess it can be merged for now. Documentation for --proxy should also be updated. FAQ entry on --proxy usage may also be useful.

Seems on Travis CI, ports in the original range are often used.

yan12125 · 2016-05-08T07:28:26Z

Some of test_socks failed on Travis CI. I guess some ports are used, so I use a different range in test_socks.py.

Documentation for --proxy should also be updated.

Done.

yan12125 · 2016-05-10T06:56:51Z

Rebased and pushed to master.

OverlordQ · 2016-05-17T04:42:12Z

Given example of socks5://127.0.0.1:1080/ fails for me.

[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--verbose', u'--proxy', u'socks5://127.0.0.1:1080', u'-f', u'bestvideo+bestaudio', u'-o', u'XXX2016.mp4', u'https://www.youtube.com/watch?v=XXXXXXX']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.05.16
[debug] Python version 2.7.11+ - Linux-4.1.0-1-amd64-x86_64-with-debian-stretch-sid
[debug] exe versions: ffmpeg 3.0.1-3, ffprobe 3.0.1-3
[debug] Proxy map: {u'http': u'socks5://127.0.0.1:1080', u'https': u'socks5://127.0.0.1:1080'}
[youtube] XXXXXXXXX: Downloading webpage
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/__init__.py", line 421, in main
    _real_main(argv)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/__init__.py", line 411, in _real_main
    retcode = ydl.download(all_urls)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 1736, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 676, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 341, in extract
    return self._real_extract(url)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/youtube.py", line 1216, in _real_extract
    video_webpage = self._download_webpage(url, video_id)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 501, in _download_webpage
    res = self._download_webpage_handle(url_or_request, video_id, note, errnote, fatal, encoding=encoding, data=data, headers=headers, query=query)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 408, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 388, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 1946, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/utils.py", line 929, in https_open
    conn_class = make_socks_conn_class(conn_class, socks_proxy)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/utils.py", line 890, in make_socks_conn_class
    compat_urllib_parse_unquote_plus(url_components.username),
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/compat.py", line 170, in compat_urllib_parse_unquote_plus
    string = string.replace('+', ' ')
AttributeError: 'NoneType' object has no attribute 'replace'

yan12125 · 2016-05-17T04:47:11Z

Sorry it's a bug caused by me. Could you try version 2016.05.10?

OverlordQ · 2016-05-17T05:45:38Z

Yeah, 2016.5.10 works fine.

Originally reported at #9287 (comment)

yan12125 · 2016-05-17T06:49:05Z

Thanks for testing. Should be fixed in cdd94c2.

Yen Chi Hsuan added 4 commits April 23, 2016 15:44

[socks] Add socks.py from @bluec0re's public domain implementation

03a0a2b

https://gist.github.com/bluec0re/cafd3764412967417fd3

[utils,compat] Move struct_pack and struct_unpack to compat.py

adc1dc9

[socks] Support SOCKS proxies

1409ad2

[test/test_socks] Add tests for SOCKS proxies

4953d30

yan12125 mentioned this pull request Apr 23, 2016

SOCKS proxy support #402

Closed

dstftw reviewed May 1, 2016
View reviewed changes

Yen Chi Hsuan added 5 commits May 3, 2016 15:11

[socks] Eliminate magic constants and improve

0ab2d37

[utils] Register SOCKS protocols in urllib and support SOCKS4A

a526eff

[utils] Add rationale for register_socks_protocols

ed8f6b7

[compat] Rename struct_(un)pack to compat_struct_(un)pack

56a1748

[test/test_socks] Test with local SOCKS servers

96edf46

dstftw reviewed May 7, 2016
View reviewed changes

Yen Chi Hsuan added 3 commits May 8, 2016 15:14

[socks] Remove a superfluous clause

47a6539

[options] Update --proxy description for SOCKS proxies

1b53d85

[test/test_socks] Use a different port range

eb87b65

Seems on Travis CI, ports in the original range are often used.

yan12125 closed this May 10, 2016

yan12125 deleted the socks-proxy branch May 10, 2016 06:57

yan12125 pushed a commit that referenced this pull request May 17, 2016

[utils] Check for None values in SOCKS proxy

cdd94c2

Originally reported at #9287 (comment)

soimort mentioned this pull request Jun 29, 2016

enable socks5 proxy like shadowsocks soimort/you-get#1231

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SOCKS proxy support #9287

SOCKS proxy support #9287

yan12125 commented Apr 23, 2016

yan12125 commented Apr 23, 2016

OliverUv commented Apr 30, 2016 •

edited

Loading

yan12125 commented Apr 30, 2016

OliverUv commented Apr 30, 2016 •

edited

Loading

OliverUv commented Apr 30, 2016

yan12125 commented Apr 30, 2016

dstftw May 1, 2016

yan12125 May 3, 2016

dstftw May 3, 2016

yan12125 May 3, 2016

OliverUv commented May 1, 2016

yan12125 commented May 1, 2016

OliverUv commented May 1, 2016

yan12125 commented May 5, 2016

dstftw May 7, 2016

dstftw commented May 7, 2016

yan12125 commented May 8, 2016

yan12125 commented May 10, 2016

OverlordQ commented May 17, 2016

yan12125 commented May 17, 2016

OverlordQ commented May 17, 2016

yan12125 commented May 17, 2016

SOCKS proxy support #9287

SOCKS proxy support #9287

Conversation

yan12125 commented Apr 23, 2016

yan12125 commented Apr 23, 2016

OliverUv commented Apr 30, 2016 • edited Loading

yan12125 commented Apr 30, 2016

OliverUv commented Apr 30, 2016 • edited Loading

OliverUv commented Apr 30, 2016

yan12125 commented Apr 30, 2016

dstftw May 1, 2016

Choose a reason for hiding this comment

yan12125 May 3, 2016

Choose a reason for hiding this comment

dstftw May 3, 2016

Choose a reason for hiding this comment

yan12125 May 3, 2016

Choose a reason for hiding this comment

OliverUv commented May 1, 2016

yan12125 commented May 1, 2016

OliverUv commented May 1, 2016

yan12125 commented May 5, 2016

dstftw May 7, 2016

Choose a reason for hiding this comment

dstftw commented May 7, 2016

yan12125 commented May 8, 2016

yan12125 commented May 10, 2016

OverlordQ commented May 17, 2016

yan12125 commented May 17, 2016

OverlordQ commented May 17, 2016

yan12125 commented May 17, 2016

OliverUv commented Apr 30, 2016 •

edited

Loading

OliverUv commented Apr 30, 2016 •

edited

Loading