[BUG] Proxy Usage for Downloading is broken #177

tophee · 2020-07-23T21:08:03Z

Blocks: #230

My 429 errors seem to be gone now, but all my downloads keep failing.

Starting with all custim arguments:

2020-07-23T20:57:50.716Z VERBOSE: youtube-dl args being used: -o,video/%(title)s.mp4,--write-info-json,--print-json,-f best[ext=mp4],--cookies,appdata/cookies.txt,--add-metadata,--no-mtime,--write-description,--embed-thumbnail
2020-07-23T20:57:51.739Z DEBUG: URL info retrieval delay: 2.231 seconds.
2020-07-23T20:57:51.739Z ERROR: Error during parsing:Error: Command failed with exit code 1: /app/node_modules/youtube-dl/bin/youtube-dl --dump-json http://www.youtube.com/watch?v=kqqGr9bYm3o
2020-07-23T20:57:51.739Z ERROR: failed to parse for urls starting with https://www.youtube.com/watch?v=kqqGr9bYm3o
2020-07-23T20:57:55.273Z ERROR: Command failed with exit code 1: /app/node_modules/youtube-dl/bin/youtube-dl --dump-json -f best -o video/%(title)s.mp4 --write-info-json --print-json -f best[ext=mp4] --cookies appdata/cookies.txt --add-metadata --no-mtime --write-description --embed-thumbnail http://www.youtube.com/watch?v=kqqGr9bYm3o

Then taking out the custom args:

2020-07-23T20:59:29.895Z VERBOSE: youtube-dl args being used: -o,video/%(title)s.mp4,--write-info-json,--print-json,-f best[ext=mp4],--cookies,appdata/cookies.txt
2020-07-23T20:59:32.086Z ERROR: Command failed with exit code 1: /app/node_modules/youtube-dl/bin/youtube-dl --dump-json -f best -o video/%(title)s.mp4 --write-info-json --print-json -f best[ext=mp4] --cookies appdata/cookies.txt http://www.youtube.com/watch?v=kqqGr9bYm3o

And then even the cookies:

2020-07-23T21:02:08.852Z VERBOSE: youtube-dl args being used: -o,video/%(title)s.mp4,--write-info-json,--print-json,-f best[ext=mp4]
2020-07-23T21:02:10.994Z ERROR: Command failed with exit code 1: /app/node_modules/youtube-dl/bin/youtube-dl --dump-json -f best -o video/%(title)s.mp4 --write-info-json --print-json -f best[ext=mp4] http://www.youtube.com/watch?v=kqqGr9bYm3o

I can download the same video with youtube-dl on my windows PC in the same network without problems...

The log settings ate on debug, so I have no idea why there's not more showing up in the log.

Edit: And yes, I did restart the container.

The text was updated successfully, but these errors were encountered:

Tzahi12345 · 2020-07-24T03:12:50Z

Can you try going into the Downloader tab in the settings and enabling "Safe download override"? I think the regular method might be bugged on some systems for some videos (worked fine on Windows, but reproduced the error on Ubuntu 18.04)

tophee · 2020-07-24T17:53:13Z

I now activated "Safe download override" but now I'm back to the 429 error:

2020-07-24T17:38:51.935Z VERBOSE: Download is running with the safe download override.
2020-07-24T17:38:51.937Z VERBOSE: youtube-dl args being used: -o,video/%(title)s.mp4,--write-info-json,--print-json,-f best[ext=mp4],--add-metadata,--no-mtime,--write-description,--embed-thumbnail
2020-07-24T17:38:57.064Z DEBUG: URL info retrieval delay: 7.073 seconds.
2020-07-24T17:38:57.065Z ERROR: Error during parsing:Error: Command failed with exit code 1: /app/node_modules/youtube-dl/bin/youtube-dl --dump-json http://www.youtube.com/watch?v=RG1iYRQtjKI
2020-07-24T17:38:57.065Z ERROR: failed to parse for urls starting with https://www.youtube.com/watch?v=RG1iYRQtjKI
2020-07-24T17:38:59.067Z DEBUG: Video download delay: 7.131 seconds.
2020-07-24T17:38:59.067Z ERROR: ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

In the mean time I have set up a VPN proxy but it doesn't work:

2020-07-24T17:47:24.458Z VERBOSE: Download is running with the safe download override.
2020-07-24T17:47:24.459Z VERBOSE: youtube-dl args being used: -o,video/%(title)s.mp4,--write-info-json,--print-json,-f best[ext=mp4],--add-metadata,--no-mtime,--write-description,--embed-thumbnail,--proxy,http://192.168.1.4:8888
2020-07-24T17:47:27.741Z DEBUG: Video download delay: 3.283 seconds.
2020-07-24T17:47:27.741Z ERROR: ERROR:

It took me a while to figure out that I need to submit the option and its arguments as two separate options: --proxy,,http://192.168.1.4:8888 but I still seem to be doing something wrong...

I have no idea where the 429s come from again. I haven't downloaded anything for about 24 hours...

Tzahi12345 · 2020-07-25T06:51:32Z

Is the proxy set up at a different public IP address? If it's the same IP it'll still be blocked. If you have access to a VPN service with SOCKS5 functionality, that would be ideal.

Anyways, I still need to solve the underlying "exit code 1" problem which seems unrelated to the 429 errors. I'll do some investigating on both ends tomorrow!

tophee · 2020-08-04T11:43:01Z

Yes, the whole point of having the proxy is that the proxy is going through a VPN. So youtube is not seeing the same IP.

Tzahi12345 · 2020-08-04T23:49:08Z

Right, just making sure. You're still getting 429's? At this point it's pretty weird it hasn't gone away..

tophee · 2020-08-05T20:24:35Z

Today I deleted the container and rebuild it with the latest image.

2020-08-05T19:37:37.646Z INFO: YoutubeDL-Material v4.0 started on PORT 17442
2020-08-05T19:37:38.081Z INFO: Found new update for youtube-dl. Updating binary...
2020-08-05T19:37:39.919Z INFO: Binary successfully updated: 2020.06.16.1 -> 2020.07.28

This is what I got when trying to download a video (without proxy):

2020-08-05T20:20:46.418Z VERBOSE: Download is running with the safe download override.
2020-08-05T20:20:46.419Z VERBOSE: youtube-dl args being used: -o,video/%(title)s.mp4,--write-info-json,--print-json,-f best[ext=mp4],--add-metadata,--no-mtime,--write-description,--embed-thumbnail
2020-08-05T20:20:46.773Z DEBUG: URL info retrieval delay: 3.255 seconds.
2020-08-05T20:20:50.822Z DEBUG: Video download delay: 4.403 seconds.
2020-08-05T20:20:50.822Z ERROR: ERROR:

Tzahi12345 · 2020-08-06T03:49:38Z

Darn youtube-dl and its useless error messages... I'm kinda lost on this one. Can you try downloading this reddit video and see if it gives you any errors? https://www.reddit.com/r/battlefield_4/comments/i4ajcd/explosive_defibrillator/

Tzahi12345 · 2020-08-10T18:31:01Z

This may be cookies messing with your downloads (old cookies caused weird errors for me), can you see if renewing them or disabling them helps?

adan89lion · 2020-08-17T09:30:38Z

I now activated "Safe download override" but now I'm back to the 429 error:

2020-07-24T17:38:51.935Z VERBOSE: Download is running with the safe download override.
2020-07-24T17:38:51.937Z VERBOSE: youtube-dl args being used: -o,video/%(title)s.mp4,--write-info-json,--print-json,-f best[ext=mp4],--add-metadata,--no-mtime,--write-description,--embed-thumbnail
2020-07-24T17:38:57.064Z DEBUG: URL info retrieval delay: 7.073 seconds.
2020-07-24T17:38:57.065Z ERROR: Error during parsing:Error: Command failed with exit code 1: /app/node_modules/youtube-dl/bin/youtube-dl --dump-json http://www.youtube.com/watch?v=RG1iYRQtjKI
2020-07-24T17:38:57.065Z ERROR: failed to parse for urls starting with https://www.youtube.com/watch?v=RG1iYRQtjKI
2020-07-24T17:38:59.067Z DEBUG: Video download delay: 7.131 seconds.
2020-07-24T17:38:59.067Z ERROR: ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

In the mean time I have set up a VPN proxy but it doesn't work:

2020-07-24T17:47:24.458Z VERBOSE: Download is running with the safe download override.
2020-07-24T17:47:24.459Z VERBOSE: youtube-dl args being used: -o,video/%(title)s.mp4,--write-info-json,--print-json,-f best[ext=mp4],--add-metadata,--no-mtime,--write-description,--embed-thumbnail,--proxy,http://192.168.1.4:8888
2020-07-24T17:47:27.741Z DEBUG: Video download delay: 3.283 seconds.
2020-07-24T17:47:27.741Z ERROR: ERROR:

It took me a while to figure out that I need to submit the option and its arguments as two separate options: --proxy,,http://192.168.1.4:8888 but I still seem to be doing something wrong...

I have no idea where the 429s come from again. I haven't downloaded anything for about 24 hours...

In regard to the proxy issue, I can confirm that --proxy flag didn't work (at least on my instance). I've set a valid proxy (where I usually use with my youtube-dl scripts), but YoutubeDL-Material is still using host connection to download videos, not proxy's (as seen in DNS requests)

Tzahi12345 · 2020-08-17T17:34:29Z

@adan89lion The proxy arg fails even with the safe download override enabled?

adan89lion · 2020-08-18T12:26:10Z

@Tzahi12345 Proxy arg works after enabling Safe Download Override. Thanks for the info.

Tzahi12345 · 2020-08-18T19:41:40Z

@adan89lion Cool! We already have a way to force safe download mode when it recognizes args that won't work the normal way, so I went ahead and added --proxy to that, see commit here.

vista-narvas · 2020-10-29T15:32:42Z

im getting the same error
using docker tzahi12345/youtubedl-material:nightly
latest was giving me trouble with the subscriptions not showing up witch worked in nightly
but now i get this error using both latest or nightly

2020-10-29T15:23:50.069Z VERBOSE: youtube-dl args being used: -o,users/admin/video/%(title)s.mp4,--write-info-json,--print-json,-f,bestvideo+bestaudio,--merge-output-format,mp4,--write-thumbnail, --proxy socks5://tor.proxy.local:9150
2020-10-29T15:23:54.898Z ERROR: Error while retrieving info on video with URL https://www.youtube.com/watch?v=92g1EhH7pHM with the following message: Error: Command failed with exit code 1: /app/node_modules/youtube-dl/bin/youtube-dl --dump-json -o users/admin/video/%(title)s.mp4 --write-info-json --print-json -f bestvideo+bestaudio --merge-output-format mp4 --write-thumbnail --proxy socks5://tor.proxy.local:9150 http://www.youtube.com/watch?v=92g1EhH7pHM

youtube doesnt like me and returns every request with a 429 error so i use a tor proxy
but even without the proxy i get error 1 and not 429

Tzahi12345 · 2020-10-31T03:16:41Z

This could simply be that youtube-dl is outdated. @vista-narvas I got the same error so at least it's reproduced. Once #234 is implemented, hopefully this will be fixed!

vista-narvas · 2020-11-02T23:21:21Z

it works
but only without the proxy args
if i use the proxy i get this error (the same error as before)
it doesnt matter if i use youtube-dl or youtube-dlc
without the proxy i would get banned in notime so i do need it

2020-11-02T22:57:23.970Z VERBOSE: youtube-dl args being used: -o,users/admin/video/%(title)s.mp4,--write-info-json,--print-json,-f,bestvideo+bestaudio,--merge-output-format,mp4,--write-thumbnail, --proxy socks5://tor.proxy.local:9150
2020-11-02T22:57:28.761Z ERROR: Error while retrieving info on video with URL https://www.youtube.com/watch?v=VAvljsMrY80 with the following message: Error: Command failed with exit code 1: /app/node_modules/youtube-dl/bin/youtube-dl --dump-json -o users/admin/video/%(title)s.mp4 --write-info-json --print-json -f bestvideo+bestaudio --merge-output-format mp4 --write-thumbnail --proxy socks5://tor.proxy.local:9150 http://www.youtube.com/watch?v=VAvljsMrY80

if i use the host name of the server for the proxy i get this error (with both youtube-dl and youtube-dlc)
(this works when i use it on my pc)

2020-11-02T23:12:06.514Z VERBOSE: youtube-dl args being used: -o,users/admin/video/%(title)s.mp4,--write-info-json,--print-json,-f,bestvideo+bestaudio,--merge-output-format,mp4,--write-thumbnail,--proxy socks://server1:9150
2020-11-02T23:12:09.084Z ERROR: Error while retrieving info on video with URL https://www.youtube.com/watch?v=J---aiyznGQ with the following message: Error: Command failed with exit code 2: /app/node_modules/youtube-dl/bin/youtube-dl --dump-json -o users/admin/video/%(title)s.mp4 --write-info-json --print-json -f bestvideo+bestaudio --merge-output-format mp4 --write-thumbnail --proxy socks://server1:9150 http://www.youtube.com/watch?v=J---aiyznGQ

its a different error failed with exit code 2

youtube-dl on my pc is version 2020.09.20 and Python version 3.8.6
for YoutubeDL-Material i use the docker nightly tag

ghost · 2020-11-03T15:49:55Z

If you use SSL inspection in your network (e.g. Untangle's SSL Inspector) and it is configured to inspect youtube traffic, try disabling the SSL inspection or add a source IP address exception. This was the resolution in my case.

vista-narvas · 2020-11-05T22:37:03Z

i think i fixed it by adding HTTP_PROXY to the environment section in my docker-compose file

    environment:
      HTTP_PROXY: "socks://tor.proxy.local:9150"
      HTTPS_PROXY: "socks://tor.proxy.local:9150"
      FTP_PROXY: "socks://tor.proxy.local:9150"

i have no way to verify if its using the proxy or normal connection
but when i disable the proxy i get an error so i think it works

Tzahi12345 · 2020-11-06T02:49:40Z

@vista-narvas Interesting.. if that's the case, then the recommended method to avoid 429 errors via proxy should be to do it in the docker-compose. Good to know!

JasonGhent · 2020-11-19T15:28:05Z

@vista-narvas I just started looking into this last night and saw similar behavior when trying to use a SOCKS proxy.

You may note in the code that there are a few places that youtubedl.exec is being called, and none of them pass in (or allow a youtubedl-material end-user to pass in) additional options to the third argument parameter, nor do they allow a way to append the --proxy argument to the downloadConfig array second argument parameter.

I think updating the service to allow this via an environment variable would be trivial, so if no one else does this before I get a chance, I'll try to put together a PR sometime later this week.

Tzahi12345 · 2020-11-20T22:56:57Z

@JasonGhent

You may note in the code that there are a few places that youtubedl.exec is being called, and none of them pass in (or allow a youtubedl-material end-user to pass in) additional options to the third argument parameter, nor do they allow a way to append the --proxy argument to the downloadConfig array second argument parameter.

You're half-right, in that with the exec method there's no dedicated parameter for proxy. Unfortunately, you can't simply pass in the proxy as it's described in the node-youtube-dl docs, because that extra param only applies to the non-exec method, and we only use the exec method.

Note that the downloadConfig is an array of args, so the current method of using a proxy is to add the proxy arg to the downloadConfig (apparently this doesn't work 100%?). This is done using the global custom args in the Downloader tab in the settings (these apply to downloads on the home page, if you want to pass custom args into subscriptions you need to use the subscription-specific custom args).

This is a little bit confusing, so if you need clarification I'll be happy to help (head over to the Discussions and open a thread there).

JasonGhent · 2020-11-21T05:52:40Z

@Tzahi12345 You're right that this can be addressed as a custom arg through the UI. I was looking at it from the perspective of a way to do it with environment variables. Mea culpa.

I did have an issue, though:

0000-00-00T00:00:00.000Z ERROR: Error while retrieving info on video with URL https://www.youtube.com/watch?v=xxxxxxxx with the following message: Error: Command failed with exit code 2: /app/node_modules/youtube-dl/bin/youtube-dl --dump-json -o video/%(title)s.mp4 --write-info-json --print-json -fbestvideo+bestaudio --merge-output-format mp4 --write-thumbnail --proxy http://127.0.0.1:1234 http://www.youtube.com/watch?v=xxxxxxxx

¯_(ツ)_/¯

Tzahi12345 · 2020-11-21T06:37:30Z

@JasonGhent No problem! I think there's several ways to go about this problem, maybe I should add a dedicated setting for it since it seems used pretty often. Or I can just tell users to use the docker-compose with Docker, or custom args without Docker.

Regarding your error, it looks like the args might be badly formed, or is this just a copy-paste error: -fbestvideo+bestaudio?

It should instead be -f bestvideo+bestaudio

Tzahi12345 added the is: bug Something isn't working label Jul 24, 2020

This was referenced Dec 1, 2020

SEVERE Performance issue #264

Closed

[RESILIENCY / SCRAPING AV] When detecting a country block, retry with Proxy. #230

Open

Tzahi12345 mentioned this issue Jan 8, 2021

Proxy doesn't work #309

Open

Tzahi12345 mentioned this issue Jan 20, 2021

HTTP Error 429: Too Many Requests #302

Closed

Tzahi12345 mentioned this issue Jan 31, 2021

--proxy arg does not work #323

Open

GlassedSilver changed the title ~~Downloads keep failing~~ [BUG] Proxy Usage for Downloading is broken May 26, 2022

GlassedSilver added the SCRAPING AUDIO/VIDEO Thing related to some DLs not downloading properly or at all. NOT for subscription-specific issues! label May 26, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Proxy Usage for Downloading is broken #177

[BUG] Proxy Usage for Downloading is broken #177

tophee commented Jul 23, 2020 •

edited by GlassedSilver

Loading

Tzahi12345 commented Jul 24, 2020 •

edited

Loading

tophee commented Jul 24, 2020 •

edited

Loading

Tzahi12345 commented Jul 25, 2020

tophee commented Aug 4, 2020

Tzahi12345 commented Aug 4, 2020

tophee commented Aug 5, 2020

Tzahi12345 commented Aug 6, 2020

Tzahi12345 commented Aug 10, 2020

adan89lion commented Aug 17, 2020

Tzahi12345 commented Aug 17, 2020

adan89lion commented Aug 18, 2020

Tzahi12345 commented Aug 18, 2020

vista-narvas commented Oct 29, 2020

Tzahi12345 commented Oct 31, 2020

vista-narvas commented Nov 2, 2020 •

edited

Loading

ghost commented Nov 3, 2020

vista-narvas commented Nov 5, 2020 •

edited

Loading

Tzahi12345 commented Nov 6, 2020

JasonGhent commented Nov 19, 2020

Tzahi12345 commented Nov 20, 2020

JasonGhent commented Nov 21, 2020 •

edited

Loading

Tzahi12345 commented Nov 21, 2020

[BUG] Proxy Usage for Downloading is broken #177

[BUG] Proxy Usage for Downloading is broken #177

Comments

tophee commented Jul 23, 2020 • edited by GlassedSilver Loading

Tzahi12345 commented Jul 24, 2020 • edited Loading

tophee commented Jul 24, 2020 • edited Loading

Tzahi12345 commented Jul 25, 2020

tophee commented Aug 4, 2020

Tzahi12345 commented Aug 4, 2020

tophee commented Aug 5, 2020

Tzahi12345 commented Aug 6, 2020

Tzahi12345 commented Aug 10, 2020

adan89lion commented Aug 17, 2020

Tzahi12345 commented Aug 17, 2020

adan89lion commented Aug 18, 2020

Tzahi12345 commented Aug 18, 2020

vista-narvas commented Oct 29, 2020

Tzahi12345 commented Oct 31, 2020

vista-narvas commented Nov 2, 2020 • edited Loading

ghost commented Nov 3, 2020

vista-narvas commented Nov 5, 2020 • edited Loading

Tzahi12345 commented Nov 6, 2020

JasonGhent commented Nov 19, 2020

Tzahi12345 commented Nov 20, 2020

JasonGhent commented Nov 21, 2020 • edited Loading

Tzahi12345 commented Nov 21, 2020

tophee commented Jul 23, 2020 •

edited by GlassedSilver

Loading

Tzahi12345 commented Jul 24, 2020 •

edited

Loading

tophee commented Jul 24, 2020 •

edited

Loading

vista-narvas commented Nov 2, 2020 •

edited

Loading

vista-narvas commented Nov 5, 2020 •

edited

Loading

JasonGhent commented Nov 21, 2020 •

edited

Loading