[cbc] Add login support

[meunierd]

Please follow the guide below

• You will be asked some questions, please read them carefully and answer honestly
• Put an x into all the boxes [ ] relevant to your pull request (like that [x])
• Use Preview tab to see how your pull request will actually look like

---

Before submitting a pull request make sure you have:

• At least skimmed through adding new extractor tutorial and youtube-dl coding conventions sections
• Searched the bugtracker for similar pull requests
• Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

• I am the original author of this code and I am willing to release it under Unlicense

What is the purpose of your pull request?

• Improvement

---

Description of your pull request and other information

_API_KEY is defined in a JS file loaded by watch.cbc.ca, it's a static value that identifies the app. You are logging into this app via login radius when you POST your account credentials.

Requests are made by a device, which registers itself and uses the provided id and token. The default token is unauthenticated but you can log the device in with the access token you receive as a response from the account login. This provides a new authenticated device token, which will let you download gated videos!

Closes #16764

[Triforcecwp]

Thanks so much for working on this @meunierd , I really appreciate it.

[srekoslav]

Can I download CBC videos with this then or it is just in testing and if so how?

[meunierd]

You can checkout the branch and use your login credentials, it works for me

…

On Fri, Feb 8, 2019 at 2:15 PM MrN00BZ ***@***.***> wrote: Can I download CBC videos with this then or it is just in testing and if so how? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#19160 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACT2TPOUz8uqWeEmhlI09AeBSxoWYH7ks5vLczJgaJpZM4amBUX> .

[srekoslav]

Sorry, but what does that mean how do you use it?

[meunierd]

If you have difficulty following the developer instructions then it's probably best to wait until this makes its way into a release

[meunierd]

@dstftw is this good to merge?

[smcclay]

I have not been able to get this to work. It's possible that I'm doing something wrong, but I have checked out this pull request and this is what I get when I run "python -m youtube_dl ..." :

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--verbose', u'-u', u'PRIVATE', u'-p', u'PRIVATE', u'https://watch.cbc.ca/media/kims-convenience/season-1/episode-1/38e815a-00ade4014dc']
[debug] Encodings: locale UTF-8, fs utf-8, out None, pref UTF-8
[debug] youtube-dl version 2019.01.30.1
[debug] Git HEAD: 7d569a37e
[debug] Python version 2.7.15 (CPython) - Darwin-18.2.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.1, ffprobe 4.1
[debug] Proxy map: {}
[debug] Using fake IP 99.238.54.128 (CA) as X-Forwarded-For.
[cbc.ca:watch] 38e815a-00ade4014dc: Downloading XML
[cbc.ca:watch] 38e815a-00ade4014dc: Downloading XML
[download] Downloading playlist: Gay Discount
[cbc.ca:watch] playlist Gay Discount: Collected 1 video ids (downloading 1 of them)
[download] Downloading video 1 of 1
[debug] Using fake IP 99.239.194.78 (CA) as X-Forwarded-For.
[cbc.ca:watch:video] ce247368-121a-4174-8b28-1879d2550f18: Downloading XML
ERROR: Unable to download XML: HTTP Error 402: Payment Required (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "youtube_dl/extractor/common.py", line 605, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "youtube_dl/YoutubeDL.py", line 2221, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/local/Cellar/python@2/2.7.15_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/local/Cellar/python@2/2.7.15_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/Cellar/python@2/2.7.15_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/local/Cellar/python@2/2.7.15_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python@2/2.7.15_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

[meunierd]

You might have an unauthenticated token in your cache, in which case it would use the cached one.

On macOS I can clear the cache by running rm ~/.cache/youtube-dl/cbcwatch/device.json

FWIW I'm able to download that episode.

[smcclay]

That fixed it for me. Thanks for doing this.

[meunierd]

@dstftw how can I move this PR forward?

[dstftw]

You must provide account credentials.

[meunierd]

username: hfkhum+ek9u6dei4bof8@sharklasers.com
password: password

[meunierd]

There we go, forgot to push the commit, sorry about that!

[makew0rld]

What happened to this, can it still be merged? Also @meunierd have you confirmed this will still work with gem.cbc.ca shows?

CC: @dstftw @remitamine

[meunierd]

Yeah I'm still using my branch. The last time I did was maybe 3 weeks ago, I pushed a rebased copy up to work with the gem.cbc.ca switch. I posted credentials, not sure what else I can do on my part.

…

On Sat, Jun 29, 2019 at 2:35 PM makeworld ***@***.***> wrote: What happened to this, can it still be merged? Also @meunierd <https://github.com/meunierd> have you confirmed this will still work with gem.cbc.ca shows? CC: @dstftw <https://github.com/dstftw> @remitamine <https://github.com/remitamine> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#19160?email_source=notifications&email_token=AAAJHWMABBEFYGHIEUI6DNLP46TQRA5CNFSM4GUYCUL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY356OQ#issuecomment-506978106>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAJHWKLEF5GT434N7P6C5TP46TQRANCNFSM4GUYCULQ> .

[makew0rld]

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://gem.cbc.ca/media/kims-convenience/season-2/episode-1/38e815a-00d01e9a6e8', '-u', 'PRIVATE', '-p', 'PRIVATE', '-v']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.06.21
[debug] Python version 3.7.3 (CPython) - Linux-5.1.12-arch1-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg 4.1.3, ffprobe 4.1.3, rtmpdump 2.4
[debug] Proxy map: {}
[debug] Using fake IP 99.254.14.117 (CA) as X-Forwarded-For.
[cbc.ca:watch] 38e815a-00d01e9a6e8: Downloading XML
[cbc.ca:watch] 38e815a-00d01e9a6e8: Downloading XML
[download] Downloading playlist: Janet's Roommate
[cbc.ca:watch] playlist Janet's Roommate: Collected 1 video ids (downloading 1 of them)
[download] Downloading video 1 of 1
[debug] Using fake IP 99.244.104.206 (CA) as X-Forwarded-For.
[cbc.ca:watch:video] 35735f55-771d-4413-add1-dacc98555036: Downloading XML
ERROR: Unable to download XML: HTTP Error 402: Payment Required (caused by <HTTPError 402: 'Payment Required'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/lib/python3.7/site-packages/youtube_dl/extractor/common.py", line 627, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 2227, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.7/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.7/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

Doesn't seem to work for me.

[ilike2burnthing]

Working fine here. Cleared .cache just to make sure. Was able to download without issue.

[meunierd]

@dstftw for a CA proxy you can use Tor with these two lines in your torrc:

ExitNodes {ca}
StrictNodes 1

Restart tor and then pass --proxy=socks5://localhost:9050 to youtube-dl.

[dstftw]

I'm well aware of that. All tor exit nodes are banned.

[meunierd]

I tested it before commenting and it works:

youtube-dl --rm-cache-dir -n --proxy=socks5://localhost:9050 'https://gem.cbc.ca/season/canada-the-story-of-us/season-1/1c9a9fb9-d1d4-4587-9f38-8c1f43d2aee0'
Removing cache dir /Users/meunierd/.cache/youtube-dl ...
[cbc.ca:watch] Downloading JSON metadata
[cbc.ca:watch] Downloading JSON metadata
[cbc.ca:watch] Acquiring device token
[cbc.ca:watch] Downloading XML
[cbc.ca:watch] 1c9a9fb9-d1d4-4587-9f38-8c1f43d2aee0: Downloading XML
[cbc.ca:watch] 1c9a9fb9-d1d4-4587-9f38-8c1f43d2aee0: Downloading XML
[download] Downloading playlist: Season 1
[cbc.ca:watch] playlist Season 1: Collected 10 video ids (downloading 10 of them)
[download] Downloading video 1 of 10
[cbc.ca:watch:video] e9923698-08b5-4209-abb3-e9fd76f4d190: Downloading XML
[cbc.ca:watch:video] e9923698-08b5-4209-abb3-e9fd76f4d190: Downloading XML
[cbc.ca:watch:video] e9923698-08b5-4209-abb3-e9fd76f4d190: Downloading m3u8 information
[download] Destination: World's Collide-38e815a-00c042d36bc.f8687.mp4
WARNING: ffmpeg does not support SOCKS proxies. Downloading is likely to fail. Consider adding --hls-prefer-native to your command.
ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
  built with Apple clang version 11.0.0 (clang-1100.0.33.17)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.2.2_2 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-libsoxr --enable-videotoolbox --disable-libjack --disable-indev=jack

EDIT: Oh I see now --

WARNING: ffmpeg does not support SOCKS proxies. Downloading is likely to fail. Consider adding --hls-prefer-native to your command.

It's just falling back to my Canadian connection, apologies.

[meunierd]

That said, I do pay for my own VPN that I could privately share. Could we exchange credentials via email?

[meunierd]

Testing with the --hls-prefer-native flag seems to respect the socks proxy:

youtube-dl --rm-cache-dir -n --hls-prefer-native --proxy=socks5://localhost:9050 'https://gem.cbc.ca/season/canada-the-story-of-us/season-1/1c9a9fb9-d1d4-4587-9f38-8c1f43d2aee0'
Removing cache dir /Users/meunierd/.cache/youtube-dl ...
[cbc.ca:watch] Downloading JSON metadata
[cbc.ca:watch] Downloading JSON metadata
[cbc.ca:watch] Acquiring device token
[cbc.ca:watch] Downloading XML
[cbc.ca:watch] 1c9a9fb9-d1d4-4587-9f38-8c1f43d2aee0: Downloading XML
[cbc.ca:watch] 1c9a9fb9-d1d4-4587-9f38-8c1f43d2aee0: Downloading XML
[download] Downloading playlist: Season 1
[cbc.ca:watch] playlist Season 1: Collected 10 video ids (downloading 10 of them)
[download] Downloading video 1 of 10
[cbc.ca:watch:video] e9923698-08b5-4209-abb3-e9fd76f4d190: Downloading XML
[cbc.ca:watch:video] e9923698-08b5-4209-abb3-e9fd76f4d190: Downloading XML
[cbc.ca:watch:video] e9923698-08b5-4209-abb3-e9fd76f4d190: Downloading m3u8 information
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 295
[download] Destination: World's Collide-38e815a-00c042d36bc.f8687.mp4
[download]   1.3% of ~2.69GiB at 575.61KiB/s ETA 01:35:00

[dstftw]

As already mentioned this does not work on web needed to verify these changed:

Also FYI geo restriction on API calls is already bypassed by using X-Forwarded-For.
My email is in profile.

[meunierd]

Okay I've emailed the proxy configs. Let me know if there's any trouble with them.

[meunierd]

I see that while the youtube-dl command succeeds, they ban basically all VPNs at the web level. They're likely using maxmind to catch anything other than residential proxies which I don't have access to. Would you be willing to tunnel through my home internet via SSH?

[dstftw]

SSH tunnel is more preferable.

[ilike2burnthing]

I'm using this outside Canada with no VPN or manual proxy, and -v shows:

[debug] Using fake IP 99.230.181.112 (CA) as X-Forwarded-For.
[cbc.ca:watch] 7d29c147-bb7e-45ac-9263-c47cab0cbe26: Downloading XML
[cbc.ca:watch] 7d29c147-bb7e-45ac-9263-c47cab0cbe26: Downloading XML
[download] Downloading playlist: Season #
[cbc.ca:watch] playlist Season #: Collected ## video ids (downloading ## of them)
[download] Downloading video 1 of ##
[debug] Using fake IP 99.248.81.57 (CA) as X-Forwarded-For.

So X-Forwarded-For seems to be doing its job fine. Apologies if I'm missing something else.

[meunierd]

Could you send me your public key and I'll get it set up.

[dstftw]

It's in ytdl downloads page.

[meunierd]

I've emailed you the SSH credentials. I tunneled through it locally myself so hopefully everything's okay.

[dstftw]

There is also a conceptual problem with cache invalidation when you try to authenticate having already cached id and token.

[dstftw]

This should be under if signature: else branch now I suppose.

[dstftw]

Are you sure signature, self._device_id will not have <, >? Strictly speaking those should be escaped or better standard XML facilities should be used for XML building.

[meunierd]

I've more clearly separated the signature/authenticated vs anonymous device registration, and escaped the signature using xml.sax.saxutils.escape from the standard library.

[meunierd]

There is also a conceptual problem with cache invalidation when you try to authenticate having already cached id and token.

Should I cache which kind of token is in cache? ie if I pass in user credentials, should it only load a signed token from cache? Your device is first registered anonymously and THEN signed so I've made a point of only setting the anonymous token when there are no login credentials present.

[dstftw]

Device id is not escaped.

[dstftw]

Caching token kind is not enough since one having already registered with some credentials may want to register under different credentials. So id and token should be cached per some checksum of particular credentials.

[makew0rld]

Does this mean main youtube-dl has CBC login support now?

[ilike2burnthing]

Does this mean main youtube-dl has CBC login support now?

Next release

EDIT: which will be ready when it's ready, just keep an eye out for it

[ilike2burnthing]

https://github.com/ytdl-org/youtube-dl/releases/tag/2020.03.24