Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Player 20dfca59 has broken YouTube extractor #32885

Closed
BloodSugar327 opened this issue Jul 30, 2024 · 14 comments · Fixed by #32891
Closed

New Player 20dfca59 has broken YouTube extractor #32885

BloodSugar327 opened this issue Jul 30, 2024 · 14 comments · Fixed by #32891
Labels
broken-IE problem with existing site extraction fixed patch-available

Comments

@BloodSugar327
Copy link

Checklist

  • [x ] I'm reporting a broken site support
  • [ x] I've verified that I'm running youtube-dl version 2024.7.25
  • [x ] I've checked that all provided URLs are alive and playable in a browser
  • [ x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
  • [ x] I've searched the bugtracker for similar issues including closed ones

Verbose log

[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2024.07.25 from yt-dlp/yt-dlp [f0993391e] (pip)
[debug] Python 3.12.3 (CPython AMD64 64bit) - Windows-11-10.0.22631-SP0 (OpenSSL 3.0.13 30 Jan 2024)
[debug] exe versions: ffmpeg N-116328-gfa5a605542-20240718 (setts), ffprobe N-116328-gfa5a605542-20240718
[debug] Optional libraries: Cryptodome-3.20.0, brotli-1.1.0, certifi-2024.07.04, mutagen-1.47.0, requests-2.32.3, sqlite3-3.45.1, urllib3-2.2.2, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1829 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: stable@2024.07.25 from yt-dlp/yt-dlp
yt-dlp is up to date (stable@2024.07.25 from yt-dlp/yt-dlp)
[youtube:tab] Extracting URL: https://music.youtube.com/playlist?list=OLAK5uy_kOExsg4A5Wc7X3_DhtK4GB0YqdeFIIjhs
WARNING: [youtube:tab] YouTube Music is not directly supported. Redirecting to https://www.youtube.com/playlist?list=OLAK5uy_kOExsg4A5Wc7X3_DhtK4GB0YqdeFIIjhs
[youtube:tab] OLAK5uy_kOExsg4A5Wc7X3_DhtK4GB0YqdeFIIjhs: Downloading webpage
[debug] [youtube:tab] Extracted SAPISID cookie
[youtube:tab] OLAK5uy_kOExsg4A5Wc7X3_DhtK4GB0YqdeFIIjhs: Redownloading playlist API JSON with unavailable videos
[download] Downloading playlist: Album - Meteora 20th Anniversary Edition
[youtube:tab] Playlist Album - Meteora 20th Anniversary Edition: Downloading 89 items of 89
[download] Downloading item 1 of 89
[youtube] Extracting URL: https://music.youtube.com/watch?v=d_iv0n9ZpAg
[youtube] d_iv0n9ZpAg: Downloading webpage
[debug] [youtube] Extracted SAPISID cookie
[youtube] d_iv0n9ZpAg: Downloading ios player API JSON
[youtube] d_iv0n9ZpAg: Downloading ios music player API JSON
[youtube] d_iv0n9ZpAg: Downloading web music client config
[youtube] d_iv0n9ZpAg: Downloading player 20dfca59
[youtube] d_iv0n9ZpAg: Downloading web music player API JSON
[debug] [youtube] Extracting signature function js_20dfca59_107
[debug] Loading youtube-sigfuncs.js_20dfca59_107 from cache
WARNING: [youtube] d_iv0n9ZpAg: nsig extraction failed: Some formats may be missing
         n = A1Ou24bVhXz1jIVy ; player = https://www.youtube.com/s/player/20dfca59/player_ias.vflset/en_US/base.js
[debug] [youtube] Unable to extract nsig function code (caused by RegexNotFoundError('Unable to extract \x1b[0;94mInitial JS player n function name\x1b[0m; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U')); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
WARNING: [youtube] d_iv0n9ZpAg: nsig extraction failed: Some formats may be missing
         n = -fLCxedkAk4LUTK2 ; player = https://www.youtube.com/s/player/20dfca59/player_ias.vflset/en_US/base.js
[debug] [youtube] Extracting signature function js_20dfca59_103
[debug] Loading youtube-sigfuncs.js_20dfca59_103 from cache
WARNING: [youtube] d_iv0n9ZpAg: nsig extraction failed: Some formats may be missing
         n = lfFDQouqFm1Pmkfk ; player = https://www.youtube.com/s/player/20dfca59/player_ias.vflset/en_US/base.js
WARNING: [youtube] d_iv0n9ZpAg: nsig extraction failed: Some formats may be missing
         n = N6FLr39qpAOmT5Ak ; player = https://www.youtube.com/s/player/20dfca59/player_ias.vflset/en_US/base.js

Description

A new player (20dfca59) has broken the YouTube extractor

@3052
Copy link

3052 commented Jul 31, 2024

this isn't YT-DLP.

@dirkf
Copy link
Contributor

dirkf commented Jul 31, 2024

Indeed: #30839: ... if you were actually running yt-dlp ...

In the UK, I'm getting 250a2ff7 which the latest yt-dl and yt-dlp handle correctly. But yt-dl will also fail with the problem player, because the way to get the n parameter from the URL is now

... a.D&&(PL(a),b=a.j.n||null)&&(b=oDa[0](b),a.set("n",b),oDa.length||rma(""))

A tweak to the extraction pattern handles this:

--- old/youtube-dl/youtube_dl/extractor/youtube.py
+++ new/youtube-dl/youtube_dl/extractor/youtube.py
     def _extract_n_function_name(self, jscode):
         func_name, idx = self._search_regex(
             # new: (b=String.fromCharCode(110),c=a.get(b))&&c=nfunc[idx](c)
-            # or:  (b="nn"[+a.D],c=a.get(b))&&(c=nfunc[idx](c)s
+            # or:  (b="nn"[+a.D],c=a.get(b))&&(c=nfunc[idx](c)
+            # or:  (PL(a),b=a.j.n||null)&&(b=nfunc[idx](b)
             # old: .get("n"))&&(b=nfunc[idx](b)
             # older: .get("n"))&&(b=nfunc(b)
             r'''(?x)
-                (?:\(\s*(?P<b>[a-z])\s*=\s*(?:
+                (?:\((?:[\w$()\s]+,)*?\s*(?P<b>[a-z])\s*=\s*(?:
                     String\s*\.\s*fromCharCode\s*\(\s*110\s*\)|
-                    "n+"\[\s*\+?s*[\w$.]+\s*]
-                )\s*,(?P<c>[a-z])\s*=\s*[a-z]\s*)?
-                \.\s*get\s*\(\s*(?(b)(?P=b)|"n{1,2}")(?:\s*\)){2}\s*&&\s*\(\s*(?(c)(?P=c)|b)\s*=\s*
+                    "n+"\[\s*\+?s*[\w$.]+\s*]|
+                    (?P<b1>(?:[\w$]+\s*\.\s*)+n\b(?:(?!&&).)+\))
+                )\s*
+                    (?(b1)
+                          &&\s*\(\s*(?P=b)|
+                          (?:
+                              ,(?P<c>[a-z])\s*=\s*[a-z]\s*)?
+                              \.\s*get\s*\(\s*(?(b)(?P=b)|"n{1,2}")(?:\s*\)){2}\s*
+                              &&\s*\(\s*(?(c)(?P=c)|(?P=b))
+                          )
+                    )\s*=\s*
                 (?P<nfunc>[a-zA-Z_$][\w$]*)(?:\s*\[(?P<idx>\d+)\])?\s*\(\s*[\w$]+\s*\)
             ''', jscode, 'Initial JS player n function name', group=('nfunc', 'idx'))
         if not idx:

@dirkf dirkf reopened this Jul 31, 2024
@dirkf
Copy link
Contributor

dirkf commented Jul 31, 2024

Also: yt-dlp/yt-dlp#10608

@GregoriusT
Copy link

GregoriusT commented Jul 31, 2024

Okay I am providing actual ytdl and ytdl-nightly logs for this, since it is the same player id.

As always, freshly compiled and ran from each repo.

==========================
TESTING NORMAL YOUTUBE-DL:
==========================


[debug] System config: []
[debug] User config: ['--no-mtime', '--match-filter', '!is_live', '--retries', 'infinite', '--fragment-retries', '3', '--skip-unavailable-fragments', '--restrict-filenames', '-i', '-o', '/home/gregorius/home/pending/videos/%(title)s___%(id)s.webm', '-f', '(bestvideo[height<=360]+worstaudio/best[height<=360])[protocol!=http_dash_segments][container!^=dash]', '--console-title', '--hls-prefer-native', '--no-cache-dir', '--http-chunk-size', '100M', '--cookies', '/home/gregorius/home/scripts/video/youtube-dl-cookies']
[debug] Custom config: []
[debug] Command-line args: ['https://www.youtube.com/watch?v=zPHM0q0xgFg', '-vf', '18', '--no-playlist', '-o', '/home/gregorius/home/scripts/video/TEST_NORMAL_%(title)s___%(id)s.webm']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Single file build
[debug] Python 3.10.12 (CPython x86_64 64bit) - Linux-5.15.0-112-generic-x86_64-with-glibc2.35 - OpenSSL 3.0.2 15 Mar 2022 - glibc 2.35
[debug] exe versions: ffmpeg 4.4.2, ffprobe 4.4.2, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] zPHM0q0xgFg: Downloading webpage
[youtube] Downloading just video zPHM0q0xgFg because of --no-playlist
[youtube] zPHM0q0xgFg: Downloading player 20dfca59
ERROR: Unable to extract nsig function code (caused by RegexNotFoundError('Unable to extract \x1b[0;34mInitial JS player n function name\x1b[0m; please report this issue on https://github.com/ytdl-org/youtube-dl/issues , using the appropriate issue template. Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose option and include the complete output.')); please report this issue on https://github.com/ytdl-org/youtube-dl/issues , using the appropriate issue template. Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose option and include the complete output.
Traceback (most recent call last):
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/extractor/youtube.py", line 1637, in _decrypt_nsig
    jsi, player_id, func_code = self._extract_n_function_code(video_id, player_url)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/extractor/youtube.py", line 1690, in _extract_n_function_code
    func_name = self._extract_n_function_name(jscode)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/extractor/youtube.py", line 1660, in _extract_n_function_name
    func_name, idx = self._search_regex(
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/extractor/common.py", line 1101, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract Initial JS player n function name; please report this issue on https://github.com/ytdl-org/youtube-dl/issues , using the appropriate issue template. Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose option and include the complete output.
Traceback (most recent call last):
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/extractor/youtube.py", line 1637, in _decrypt_nsig
    jsi, player_id, func_code = self._extract_n_function_code(video_id, player_url)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/extractor/youtube.py", line 1690, in _extract_n_function_code
    func_name = self._extract_n_function_name(jscode)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/extractor/youtube.py", line 1660, in _extract_n_function_name
    func_name, idx = self._search_regex(
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/extractor/common.py", line 1101, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract Initial JS player n function name; please report this issue on https://github.com/ytdl-org/youtube-dl/issues , using the appropriate issue template. Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose option and include the complete output.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/YoutubeDL.py", line 875, in wrapper
    return func(self, *args, **kwargs)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/YoutubeDL.py", line 971, in __extract_info
    ie_result = ie.extract(url)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/extractor/common.py", line 571, in extract
    ie_result = self._real_extract(url)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/extractor/youtube.py", line 2108, in _real_extract
    self._unthrottle_format_urls(video_id, player_url, dct)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/extractor/youtube.py", line 1725, in _unthrottle_format_urls
    n_response = decrypt_nsig(n_param)(n_param, video_id, player_url)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/extractor/youtube.py", line 1614, in inner
    raise ret
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/extractor/youtube.py", line 1606, in inner
    self._player_cache[cache_id] = func(*args, **kwargs)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/extractor/youtube.py", line 1639, in _decrypt_nsig
    raise ExtractorError('Unable to extract nsig function code', cause=e)
youtube_dl.utils.ExtractorError: Unable to extract nsig function code (caused by RegexNotFoundError('Unable to extract \x1b[0;34mInitial JS player n function name\x1b[0m; please report this issue on https://github.com/ytdl-org/youtube-dl/issues , using the appropriate issue template. Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose option and include the complete output.')); please report this issue on https://github.com/ytdl-org/youtube-dl/issues , using the appropriate issue template. Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose option and include the complete output.


===========================
TESTING NIGHTLY YOUTUBE-DL:
===========================


[debug] System config: []
[debug] User config: ['--no-mtime', '--match-filter', '!is_live', '--retries', 'infinite', '--fragment-retries', '3', '--skip-unavailable-fragments', '--restrict-filenames', '-i', '-o', '/home/gregorius/home/pending/videos/%(title)s___%(id)s.webm', '-f', '(bestvideo[height<=360]+worstaudio/best[height<=360])[protocol!=http_dash_segments][container!^=dash]', '--console-title', '--hls-prefer-native', '--no-cache-dir', '--http-chunk-size', '100M', '--cookies', '/home/gregorius/home/scripts/video/youtube-dl-cookies']
[debug] Custom config: []
[debug] Command-line args: ['https://www.youtube.com/watch?v=zPHM0q0xgFg', '-vf', '18', '--no-playlist', '-o', '/home/gregorius/home/scripts/video/TEST_NIGHTLY_%(title)s___%(id)s.webm']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2024.07.25 [e1b3fa242] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
[debug] Python 3.10.12 (CPython x86_64 64bit) - Linux-5.15.0-112-generic-x86_64-with-glibc2.35 - OpenSSL 3.0.2 15 Mar 2022 - glibc 2.35
[debug] exe versions: ffmpeg 4.4.2, ffprobe 4.4.2, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] zPHM0q0xgFg: Downloading webpage
[youtube] Downloading just video zPHM0q0xgFg because of --no-playlist
[youtube] zPHM0q0xgFg: Downloading player 20dfca59
ERROR: Unable to extract nsig function code (caused by RegexNotFoundError('Unable to extract \x1b[0;34mInitial JS player n function name\x1b[0m; please report this issue on https://github.com/ytdl-org/youtube-dl/issues , using the appropriate issue template. Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose option and include the complete output.')); please report this issue on https://github.com/ytdl-org/youtube-dl/issues , using the appropriate issue template. Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose option and include the complete output.
Traceback (most recent call last):
  File "/home/gregorius/home/scripts/video/youtube-dl-nightly/youtube-dl/youtube_dl/extractor/youtube.py", line 1637, in _decrypt_nsig
    jsi, player_id, func_code = self._extract_n_function_code(video_id, player_url)
  File "/home/gregorius/home/scripts/video/youtube-dl-nightly/youtube-dl/youtube_dl/extractor/youtube.py", line 1690, in _extract_n_function_code
    func_name = self._extract_n_function_name(jscode)
  File "/home/gregorius/home/scripts/video/youtube-dl-nightly/youtube-dl/youtube_dl/extractor/youtube.py", line 1660, in _extract_n_function_name
    func_name, idx = self._search_regex(
  File "/home/gregorius/home/scripts/video/youtube-dl-nightly/youtube-dl/youtube_dl/extractor/common.py", line 1101, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract Initial JS player n function name; please report this issue on https://github.com/ytdl-org/youtube-dl/issues , using the appropriate issue template. Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose option and include the complete output.
Traceback (most recent call last):
  File "/home/gregorius/home/scripts/video/youtube-dl-nightly/youtube-dl/youtube_dl/extractor/youtube.py", line 1637, in _decrypt_nsig
    jsi, player_id, func_code = self._extract_n_function_code(video_id, player_url)
  File "/home/gregorius/home/scripts/video/youtube-dl-nightly/youtube-dl/youtube_dl/extractor/youtube.py", line 1690, in _extract_n_function_code
    func_name = self._extract_n_function_name(jscode)
  File "/home/gregorius/home/scripts/video/youtube-dl-nightly/youtube-dl/youtube_dl/extractor/youtube.py", line 1660, in _extract_n_function_name
    func_name, idx = self._search_regex(
  File "/home/gregorius/home/scripts/video/youtube-dl-nightly/youtube-dl/youtube_dl/extractor/common.py", line 1101, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract Initial JS player n function name; please report this issue on https://github.com/ytdl-org/youtube-dl/issues , using the appropriate issue template. Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose option and include the complete output.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/gregorius/home/scripts/video/youtube-dl-nightly/youtube-dl/youtube_dl/YoutubeDL.py", line 879, in wrapper
    return func(self, *args, **kwargs)
  File "/home/gregorius/home/scripts/video/youtube-dl-nightly/youtube-dl/youtube_dl/YoutubeDL.py", line 975, in __extract_info
    ie_result = ie.extract(url)
  File "/home/gregorius/home/scripts/video/youtube-dl-nightly/youtube-dl/youtube_dl/extractor/common.py", line 571, in extract
    ie_result = self._real_extract(url)
  File "/home/gregorius/home/scripts/video/youtube-dl-nightly/youtube-dl/youtube_dl/extractor/youtube.py", line 2108, in _real_extract
    self._unthrottle_format_urls(video_id, player_url, dct)
  File "/home/gregorius/home/scripts/video/youtube-dl-nightly/youtube-dl/youtube_dl/extractor/youtube.py", line 1725, in _unthrottle_format_urls
    n_response = decrypt_nsig(n_param)(n_param, video_id, player_url)
  File "/home/gregorius/home/scripts/video/youtube-dl-nightly/youtube-dl/youtube_dl/extractor/youtube.py", line 1614, in inner
    raise ret
  File "/home/gregorius/home/scripts/video/youtube-dl-nightly/youtube-dl/youtube_dl/extractor/youtube.py", line 1606, in inner
    self._player_cache[cache_id] = func(*args, **kwargs)
  File "/home/gregorius/home/scripts/video/youtube-dl-nightly/youtube-dl/youtube_dl/extractor/youtube.py", line 1639, in _decrypt_nsig
    raise ExtractorError('Unable to extract nsig function code', cause=e)
youtube_dl.utils.ExtractorError: Unable to extract nsig function code (caused by RegexNotFoundError('Unable to extract \x1b[0;34mInitial JS player n function name\x1b[0m; please report this issue on https://github.com/ytdl-org/youtube-dl/issues , using the appropriate issue template. Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose option and include the complete output.')); please report this issue on https://github.com/ytdl-org/youtube-dl/issues , using the appropriate issue template. Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose option and include the complete output.

@BloodSugar327
Copy link
Author

Also: yt-dlp/yt-dlp#10608

Thank you, so sorry about that. The referenced branch fixes what you've mentioned. I really appreciate the help!

@aiur-adept
Copy link
Contributor

@dirkf is there a branch/release with the nsig fix for youtube-dl, or just for yt-dlp? Should we take your patch above and put it in a PR?

@dirkf
Copy link
Contributor

dirkf commented Aug 1, 2024

Actually, a PR would be very helpful as I'm busy on something else. I have some further tweaks but I can add those to the patch above.

We need the matching test too:

--- old/youtube-dl/test/test_youtube_signature.py
+++ new/youtube-dl/test/test_youtube_signature.py
@@ -173,6 +173,10 @@
     (
         'https://www.youtube.com/s/player/5604538d/player_ias.vflset/en_US/base.js',
         '7X-he4jjvMx7BCX', 'sViSydX8IHtdWA',
+    ),
+    (
+        'https://www.youtube.com/s/player/20dfca59/player_ias.vflset/en_US/base.js',
+        '-fLCxedkAk4LUTK2', 'O8kfRq1y1eyHGw',
     ),
 ]
 

@aiur-adept
Copy link
Contributor

@dirkf for the nsig fix patch, i'm getting **** Only garbage was found in the patch input. How should I apply it, cause I guess whatever i'm doing isn't working (tried a few different ways of invoking patch).

@dirkf
Copy link
Contributor

dirkf commented Aug 1, 2024

I'd just edit the function on the first line (ie, removing the - lines and adding the + lines). The test patch should apply properly with a suitable -p ... value.

You can make one commit or more, as you like. I'll probably merge the PR into a single commit.

@aiur-adept
Copy link
Contributor

For some reason it just wouldn't apply with any value of -p... Just did it by hand, and checked that it works to download the playlist OP saw broken:

https://www.youtube.com/playlist?list=OLAK5uy_kOExsg4A5Wc7X3_DhtK4GB0YqdeFIIjhs

It works, but then when I ran the test_youtube_signature.py, I saw

Ran 37 tests in 26.273s

FAILED (errors=23)

With lots of Unable to extract Initial JS player n function name. Are these failures expected?

@dirkf dirkf linked a pull request Aug 1, 2024 that will close this issue
11 tasks
@dirkf dirkf added broken-IE problem with existing site extraction patch-available fixed labels Aug 1, 2024
@GregoriusT
Copy link

GregoriusT commented Aug 1, 2024

New Error that i get kindof consistently now after this fix, not 100% sure what went wrong there. Will open a new Issue if necessary.

Edit: I should mention Format 18 (the super basic one) works, everything else does not because Error 403 showing up 95% of the time.

==========================
TESTING NORMAL YOUTUBE-DL:
==========================


[debug] System config: []
[debug] User config: ['--no-mtime', '--match-filter', '!is_live', '--retries', 'infinite', '--fragment-retries', '3', '--skip-unavailable-fragments', '--restrict-filenames', '-i', '-o', '/home/gregorius/home/pending/videos/%(title)s___%(id)s.webm', '-f', '(bestvideo[height<=360]+worstaudio/best[height<=360])[protocol!=http_dash_segments][container!^=dash]', '--console-title', '--hls-prefer-native', '--no-cache-dir', '--http-chunk-size', '100M', '--cookies', '/home/gregorius/home/scripts/video/youtube-dl-cookies']
[debug] Custom config: []
[debug] Command-line args: ['https://www.youtube.com/watch?v=RfD1fOoRl3s', '-vf', '(242+249/242+250/242+171/242+251)/(243+249/243+250/243+171/243+251)/18', '--no-playlist', '-o', '/home/gregorius/home/scripts/video/TEST_NORMAL_%(title)s___%(id)s.webm']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Single file build
[debug] Python 3.10.12 (CPython x86_64 64bit) - Linux-5.15.0-112-generic-x86_64-with-glibc2.35 - OpenSSL 3.0.2 15 Mar 2022 - glibc 2.35
[debug] exe versions: ffmpeg 4.4.2, ffprobe 4.4.2, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] RfD1fOoRl3s: Downloading webpage
[youtube] Downloading just video RfD1fOoRl3s because of --no-playlist
[youtube] RfD1fOoRl3s: Downloading player d2e656ee
[debug] [youtube] Decrypted nsig 2ML9SYny8izArpmO => GVDMNIwc2DAVvw
[debug] [youtube] Decrypted nsig h4XxQcSQWAgd7VSV => xJzN3kJeyXrp5g
[debug] Invoking downloader on 'https://rr1---sn-4g5lznle.googlevideo.com/videoplayback?expire=1722561354&ei=6t6rZsSsNcCgi9oPuPmh-Ak&ip=47.69.169.114&id=o-AOrpWcupmpafDRT5jJ0aaqwXhaN7g3hVRoFLEi9kVq7e&itag=242&aitags=133%2C134%2C135%2C136%2C160%2C242%2C243%2C244%2C247%2C278%2C298%2C299%2C302%2C303%2C308%2C315&source=youtube&requiressl=yes&xpc=EgVo2aDSNQ%3D%3D&mh=6T&mm=31%2C26&mn=sn-4g5lznle%2Csn-f5f7kn7z&ms=au%2Conr&mv=m&mvi=1&pl=19&initcwndbps=1712500&bui=AXc671JFI7apstfWNpNKxfyUI0nkQlksUnUgILpmuSAOhoUyOpQmIWR4RCqULf2XuM0ozMlPPpHqm8mm&spc=NO7bAcs8fS6B4X9himXdxQ-F8sOGi4LbKfr8AC327rlaUbRR7VN3_JTnl3SA&vprv=1&svpuc=1&mime=video%2Fwebm&ns=PZSanBQxVeaKXJ0XUpXp5ngQ&rqh=1&gir=yes&clen=38439826&dur=2650.748&lmt=1721863622832436&mt=1722539374&fvip=2&keepalive=yes&c=WEB&sefc=1&txp=4437434&n=xJzN3kJeyXrp5g&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cxpc%2Cbui%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cns%2Crqh%2Cgir%2Cclen%2Cdur%2Clmt&sig=AJfQdSswRAIgdx4cDe-zP8k_7IjdNrtadYtndWX7grzveajCboHnyigCIHn-z-OAkc08GHlxov8MxxsdxWQBOMdlUQfjSPEd9khS&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AGtxev0wRAIgE9Ku0qDkD7j1s7Y7MBg1wl-cYPhz-u9ULg__zMDz7UECICZji3gl3QrmpVVq4i-2Ms0TkGWIJ8AJNg9lYlZh7kuO'
[dashsegments] Total fragments: 4
[download] Destination: /home/gregorius/home/scripts/video/TEST_NORMAL_Life_As_A_Planetary_Governor___RfD1fOoRl3s.f242.webm
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 1 of 3)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 2 of 3)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 3 of 3)...
ERROR: giving up after 3 fragment retries
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/__main__.py", line 19, in <module>
    youtube_dl.main()
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/__init__.py", line 477, in main
    _real_main(argv)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/__init__.py", line 467, in _real_main
    retcode = ydl.download(all_urls)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/YoutubeDL.py", line 2253, in download
    res = self.extract_info(
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/YoutubeDL.py", line 868, in extract_info
    return self.__extract_info(url, ie, download, extra_info, process)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/YoutubeDL.py", line 875, in wrapper
    return func(self, *args, **kwargs)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/YoutubeDL.py", line 982, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/YoutubeDL.py", line 1016, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/YoutubeDL.py", line 1860, in process_video_result
    self.process_info(new_info)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/YoutubeDL.py", line 138, in wrapper
    return func(self, *args, **kwargs)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/YoutubeDL.py", line 2152, in process_info
    partial_success = dl(fname, new_info)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/YoutubeDL.py", line 2088, in dl
    return fd.download(name, new_info)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/downloader/common.py", line 380, in download
    return self.real_download(filename, info_dict)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/downloader/dash.py", line 78, in real_download
    self.report_error('giving up after %s fragment retries' % count)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/downloader/common.py", line 175, in report_error
    self.ydl.report_error(*args, **kargs)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/YoutubeDL.py", line 678, in report_error
    self.trouble(*args, **kwargs)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/YoutubeDL.py", line 630, in trouble
    tb_data = traceback.format_list(traceback.extract_stack())

[debug] Invoking downloader on 'https://rr1---sn-4g5lznle.googlevideo.com/videoplayback?expire=1722561354&ei=6t6rZsSsNcCgi9oPuPmh-Ak&ip=47.69.169.114&id=o-AOrpWcupmpafDRT5jJ0aaqwXhaN7g3hVRoFLEi9kVq7e&itag=249&source=youtube&requiressl=yes&xpc=EgVo2aDSNQ%3D%3D&mh=6T&mm=31%2C26&mn=sn-4g5lznle%2Csn-f5f7kn7z&ms=au%2Conr&mv=m&mvi=1&pl=19&initcwndbps=1712500&bui=AXc671JFI7apstfWNpNKxfyUI0nkQlksUnUgILpmuSAOhoUyOpQmIWR4RCqULf2XuM0ozMlPPpHqm8mm&spc=NO7bAcs8fS6B4X9himXdxQ-F8sOGi4LbKfr8AC327rlaUbRR7VN3_JTnl3SA&vprv=1&svpuc=1&mime=audio%2Fwebm&ns=PZSanBQxVeaKXJ0XUpXp5ngQ&rqh=1&gir=yes&clen=16936580&dur=2650.781&lmt=1721832658250847&mt=1722539374&fvip=2&keepalive=yes&c=WEB&sefc=1&txp=4432434&n=xJzN3kJeyXrp5g&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cxpc%2Cbui%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cns%2Crqh%2Cgir%2Cclen%2Cdur%2Clmt&sig=AJfQdSswRQIhANfl9S6HKYwrUMbjJ9oh5BTm29MzrWOCfVYUn6vjHhbYAiB4f2whhIxBP9OgR102xKimRPYQnhxyHiQy042oS3waIA%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AGtxev0wRAIgE9Ku0qDkD7j1s7Y7MBg1wl-cYPhz-u9ULg__zMDz7UECICZji3gl3QrmpVVq4i-2Ms0TkGWIJ8AJNg9lYlZh7kuO'
[dashsegments] Total fragments: 2
[download] Destination: /home/gregorius/home/scripts/video/TEST_NORMAL_Life_As_A_Planetary_Governor___RfD1fOoRl3s.f249.webm
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 1 of 3)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 2 of 3)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 3 of 3)...
ERROR: giving up after 3 fragment retries
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/__main__.py", line 19, in <module>
    youtube_dl.main()
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/__init__.py", line 477, in main
    _real_main(argv)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/__init__.py", line 467, in _real_main
    retcode = ydl.download(all_urls)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/YoutubeDL.py", line 2253, in download
    res = self.extract_info(
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/YoutubeDL.py", line 868, in extract_info
    return self.__extract_info(url, ie, download, extra_info, process)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/YoutubeDL.py", line 875, in wrapper
    return func(self, *args, **kwargs)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/YoutubeDL.py", line 982, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/YoutubeDL.py", line 1016, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/YoutubeDL.py", line 1860, in process_video_result
    self.process_info(new_info)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/YoutubeDL.py", line 138, in wrapper
    return func(self, *args, **kwargs)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/YoutubeDL.py", line 2152, in process_info
    partial_success = dl(fname, new_info)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/YoutubeDL.py", line 2088, in dl
    return fd.download(name, new_info)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/downloader/common.py", line 380, in download
    return self.real_download(filename, info_dict)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/downloader/dash.py", line 78, in real_download
    self.report_error('giving up after %s fragment retries' % count)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/downloader/common.py", line 175, in report_error
    self.ydl.report_error(*args, **kargs)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/YoutubeDL.py", line 678, in report_error
    self.trouble(*args, **kwargs)
  File "/home/gregorius/home/scripts/video/youtube-dl-normal/youtube-dl/youtube_dl/YoutubeDL.py", line 630, in trouble
    tb_data = traceback.format_list(traceback.extract_stack())

@dirkf
Copy link
Contributor

dirkf commented Aug 2, 2024

See #32882

@GregoriusT
Copy link

minor question, has youtube always had the "sign in to prove you're not a bot" thing? because after the fix this particular error happens with my vpn too now (but not an issue without that particular selfhosted vpn), and i do not know whether that would qualify as a bug or not here, since i am pretty sure you can give ytdl your account info.

@dirkf
Copy link
Contributor

dirkf commented Aug 2, 2024

See yt-dlp discussion here and in linked issues. Maybe staying under the radar is the answer, though a known data centre IP might be targeted regardless.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
broken-IE problem with existing site extraction fixed patch-available
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants