Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Science Channel doesn't use passed cookies #21980

Closed
5 tasks done
Diegus83 opened this issue Aug 2, 2019 · 2 comments
Closed
5 tasks done

Science Channel doesn't use passed cookies #21980

Diegus83 opened this issue Aug 2, 2019 · 2 comments
Assignees
@remitamine
Labels
regression tv-provider-account-needed

Comments

@Diegus83
Copy link

Diegus83 commented Aug 2, 2019 •
edited
Loading

Checklist

  • I'm reporting a broken site support
  • I've verified that I'm running youtube-dl version 2019.08.02
  • I've checked that all provided URLs are alive and playable in a browser
  • I've checked that all URLs and arguments with special characters are properly quoted or escaped
  • I've searched the bugtracker for similar issues including closed ones

Verbose log

/usr/local/bin/python3 /usr/local/bin/youtube-dl  https://www.sciencechannel.com/tv-shows/mythbusters-on-science/full-episodes/christmas-special --cookies cookies.txt --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://www.sciencechannel.com/tv-shows/mythbusters-on-science/full-episodes/christmas-special', '--cookies', 'cookies.txt', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.08.02
[debug] Python version 3.7.4 (CPython) - Darwin-18.7.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.1.4, ffprobe 4.1.4, rtmpdump 2.4
[debug] Proxy map: {}
[Discovery] christmas-special: Downloading content JSON metadata
[Discovery] christmas-special: Downloading streaming JSON metadata
ERROR: This video is only available via cable service provider subscription that is not currently supported. You may want to use --cookies.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 627, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2227, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/discovery.py", line 98, in _real_extract
    display_id, 'Downloading streaming JSON metadata', headers=headers)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 892, in _download_json
    expected_status=expected_status)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 870, in _download_json_handle
    expected_status=expected_status)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 660, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 645, in _request_webpage
    raise ExtractorError(errmsg, sys.exc_info()[2], cause=err)
youtube_dl.utils.ExtractorError: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 627, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2227, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/discovery.py", line 98, in _real_extract
    display_id, 'Downloading streaming JSON metadata', headers=headers)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 892, in _download_json
    expected_status=expected_status)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 870, in _download_json_handle
    expected_status=expected_status)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 660, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 645, in _request_webpage
    raise ExtractorError(errmsg, sys.exc_info()[2], cause=err)
youtube_dl.utils.ExtractorError: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 796, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 530, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/discovery.py", line 108, in _real_extract
    ' is not currently supported. You may want to use --cookies.', expected=True)
youtube_dl.utils.ExtractorError: This video is only available via cable service provider subscription that is not currently supported. You may want to use --cookies.

Description

I tried other videos on the Science Channel that don't require login and they work fine, but for videos that require login, passing --cookies results in a 403 Error and the recommendation to use --cookies.

I also tried some go.discovery.com URLs with the same cookie file and it is working properly, so the cookie files is well formed.

The extractor works fine in version 2019-07-30 (see more logs below) so the problem may be caused by some of the latest changes to the discovery extractor.
@dstftw dstftw closed this as completed Aug 2, 2019
@Diegus83
Copy link
Author

Diegus83 commented Aug 2, 2019 •
edited
Loading

Going back to youtube-dl 2019-07-16 fixes the issue so the problem may be caused by some change introduced recently on the code and not changes on the website

python3 -m youtube_dl --cookies ../cookies.txt https://www.sciencechannel.com/tv-shows/mythbusters-on-science/full-episodes/christmas-special --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--cookies', '../cookies.txt', 'https://www.sciencechannel.com/tv-shows/mythbusters-on-science/full-episodes/christmas-special', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.07.16
[debug] Python version 3.7.4 (CPython) - Darwin-18.7.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.1.4, ffprobe 4.1.4, rtmpdump 2.4
[debug] Proxy map: {}
[Discovery] christmas-special: Downloading webpage
[Discovery] christmas-special: Downloading JSON metadata
[Discovery] christmas-special: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://content-ause1.uplynk.com/fabe205a6b3949a4b0f8f2550091e566/i.m3u8?tc=1&exp=1564756190&rn=2094315310&ct=a&cid=fabe205a6b3949a4b0f8f2550091e566&ad.pingf=3&pp2ip=0&ad.cping=1&ad=fw&rays=cdefghiba&v=2&ad.customer_id=&ad.nw=&ad.prof=&ad.csid=&ad.vip=69.116.155.129&ap.use=0&sig=7d8bb6e39170ec82031e7c599b2811e17c6e099a0aaa5c9cde03764fcb2a25f6&pbs=0e2ec85695e141bbb699519191b80b32'
[hlsnative] Downloading m3u8 manifest
WARNING: hlsnative has detected features it does not support, extraction will be delegated to ffmpeg
[download] Destination: Christmas Special-59c559d16b66d1704a6c2b79.mp4

Also works on 2019-07-30

python3 -m youtube_dl --cookies ../cookies.txt https://www.sciencechannel.com/tv-shows/mythbusters-on-science/full-episodes/christmas-special --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--cookies', '../cookies.txt', 'https://www.sciencechannel.com/tv-shows/mythbusters-on-science/full-episodes/christmas-special', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.07.30
[debug] Python version 3.7.4 (CPython) - Darwin-18.7.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.1.4, ffprobe 4.1.4, rtmpdump 2.4
[debug] Proxy map: {}
[Discovery] christmas-special: Downloading webpage
[Discovery] christmas-special: Downloading JSON metadata
[Discovery] christmas-special: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://content-ausc3.uplynk.com/fabe205a6b3949a4b0f8f2550091e566/i.m3u8?tc=1&exp=1564756531&rn=1341401075&ct=a&cid=fabe205a6b3949a4b0f8f2550091e566&ad.pingf=3&pp2ip=0&ad.cping=1&ad=fw&rays=cdefghiba&v=2&ad.customer_id=&ad.nw=&ad.prof=&ad.csid=&ad.vip=69.116.155.129&ap.use=0&sig=fa9d6f0dc1b890c3a21d720a855a3edb3bd55f9cec8a8330eae83e9aad4c945a&pbs=6ed806a88c7b41de8161705c9a3b431e'

Diegus83 referenced this issue in Vikash-Kothary/youtube-dl Aug 2, 2019
@remitamine
[discovery] use API call for video data extraction(#21808)
5351116
@remitamine remitamine reopened this Aug 2, 2019
@remitamine remitamine self-assigned this Aug 2, 2019
@remitamine
Copy link
Collaborator

will be fixed in the next version:

[Discovery] christmas-special: Downloading content JSON metadata
[Discovery] christmas-special: Downloading streaming JSON metadata
[Discovery] christmas-special: Downloading m3u8 information
[download] Destination: Christmas Special-59c559d16b66d1704a6c2b79.mp4
ffmpeg version n4.1.4 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 9.1.0 (GCC)
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-nvdec --enable-nvenc --enable-omx --enable-shared --enable-version3
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100
  libpostproc    55.  3.100 / 55.  3.100
[hls,applehttp @ 0x55edef584180] Opening 'https://content-ause4.uplynk.com/check2?b=fabe205a6b3949a4b0f8f2550091e566&v=fabe205a6b3949a4b0f8f2550091e566&r=i&pbs=3115e887b2db4783aa01afee1c3f2f07' for reading
[hls,applehttp @ 0x55edef584180] Opening 'crypto+https://x-default-stgec.uplynk.com/ausw/slices/fab/e6cf0c55dac249f0a0f72e7c72e6f6cb/fabe205a6b3949a4b0f8f2550091e566/I00000000.ts?pbs=3115e887b2db4783aa01afee1c3f2f07&si=0' for reading
Input #0, hls,applehttp, from 'https://content-ause4.uplynk.com/fabe205a6b3949a4b0f8f2550091e566/i.m3u8?tc=1&exp=1564766038&rn=830599786&ct=a&cid=fabe205a6b3949a4b0f8f2550091e566&ad.pingf=3&pp2ip=0&ad.cping=1&ad=fw&rays=cdefghiba&v=2&ad.customer_id=&ad.nw=&ad.prof=&ad.csid=&ad.vip=193.37.254.138&ap.use=0&sig=66d73e8d96f878c064beeb57d57acbe11c5bc67b9c1bcd70fd3ca39655ad69e5&pbs=3115e887b2db4783aa01afee1c3f2f07':
  Duration: 00:42:46.10, start: 0.000000, bitrate: 0 kb/s
  Program 0 
    Metadata:
      variant_bitrate : 0
    Stream #0:0: Audio: aac (HE-AAC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp
    Metadata:
      variant_bitrate : 0
    Stream #0:1: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 640x480, 29.97 fps, 29.97 tbr, 90k tbn, 60 tbc
    Metadata:
      variant_bitrate : 0
    Stream #0:2: Data: timed_id3 (ID3  / 0x20334449)
    Metadata:
      variant_bitrate : 0
Output #0, mp4, to 'file:Christmas Special-59c559d16b66d1704a6c2b79.mp4.part':
  Metadata:
    encoder         : Lavf58.20.100
    Stream #0:0: Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480, q=2-31, 29.97 fps, 29.97 tbr, 90k tbn, 90k tbc
    Metadata:
      variant_bitrate : 0
    Stream #0:1: Audio: aac (HE-AAC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp
    Metadata:
      variant_bitrate : 0
Stream mapping:
  Stream #0:1 -> #0:0 (copy)
  Stream #0:0 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame=   28 fps=0.0 q=-1.0 size=     256kB time=00:00:00.93 bitrate=2234.6kbits/frame=   37 fps= 25 q=-1.0 size=     512kB time=00:00:01.28 bitrate=3277.1kbits/frame=   44 fps= 21 q=-1.0 size=     512kB time=00:00:01.49 bitrate=2808.9kbits/frame=   51 fps= 19 q=-1.0 size=     768kB time=00:00:01.74 bitrate=3596.7kbits/frame=   54 fps= 16 q=-1.0 size=     768kB time=00:00:01.83 bitrate=3429.4kbits/frame=   63 fps= 16 q=-1.0 size=    1024kB time=00:00:02.13 bitrate=3932.3kbits/^Cframe=   69 fps= 15 q=-1.0 Lsize=    1208kB time=00:00:02.34 bitrate=4217.3kbits/s speed=0.513x    
video:1168kB audio:37kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.217354%
Exiting normally, received signal 2.

ERROR: Interrupted by user

Lamieur referenced this issue in Lamieur/youtube-dl Aug 3, 2019
@remitamine @Lamieur
[discovery] limit video data by show slug(closes #21980)
d119c8f
meunierd referenced this issue in meunierd/youtube-dl Feb 13, 2020
@remitamine @meunierd
[discovery] limit video data by show slug(closes #21980)
2b6891e
Lamieur referenced this issue in Lamieur/youtube-dl Apr 20, 2020
@Lamieur
Revert "[discovery] limit video data by show slug(closes #21980)"
a15811f 
This reverts commit d119c8f.
Lamieur referenced this issue in Lamieur/youtube-dl Apr 20, 2020
@Lamieur
Revert "[discovery] limit video data by show slug(closes #21980)"
b49ed5e 
This reverts commit d119c8f.
Lamieur referenced this issue in Lamieur/youtube-dl Apr 20, 2020
@Lamieur
Revert "Revert "[discovery] limit video data by show slug(closes #219…
7171198 
…80)""

This reverts commit b49ed5e.
Lamieur referenced this issue in Lamieur/youtube-dl Apr 20, 2020
@Lamieur
Revert "[discovery] limit video data by show slug(closes #21980)"
c57bf55 
This reverts commit d119c8f.
pareronia referenced this issue in pareronia/youtube-dl Jun 22, 2020
@remitamine @pareronia
[discovery] limit video data by show slug(closes #21980)
ae541ef
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@remitamine remitamine

Labels
regression tv-provider-account-needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dstftw @remitamine @Diegus83