Better handling of persistent errors such as 429

[tophee]

My downloads are failing again. This is what I keep getting:

2020-07-19T14:05:29.553Z DEBUG: URL info retrieval delay: 2.103 seconds.
2020-07-19T14:05:29.553Z ERROR: Error during parsing:Error: Command failed with exit code 1: /app/node_modules/youtube-dl/bin/youtube-dl --dump-json http://www.youtube.com/watch?v=k9tHJPpew5s
2020-07-19T14:05:29.553Z ERROR: failed to parse for urls starting with https://www.youtube.com/watch?v=k9tHJPpew5s
2020-07-19T14:05:30.679Z VERBOSE: youtube-dl args being used: -o,video/%(title)s.mp4,--write-info-json,--print-json,-f best[ext=mp4],--add-metadata,--no-mtime,--write-description,--embed-thumbnail
2020-07-19T14:05:32.883Z ERROR: Command failed with exit code 1: /app/node_modules/youtube-dl/bin/youtube-dl --dump-json -f best -o video/%(title)s.mp4 --write-info-json --print-json -f best[ext=mp4] --add-metadata --no-mtime --write-description --embed-thumbnail http://www.youtube.com/watch?v=k9tHJPpew5s

I have tried removing all custom arguments and I even created an API-key and told yt-dl material to use it but to no avail.

[Tzahi12345]

Weird, it works on my end, even with the additional args:

Exit code 1 signifies a syntax error, but there is nothing wrong with the command that was ultimately sent. Do you get this error with any other videos?

[tophee]

I get this error on any video. I don't know if this is related, but I am also having trouble with another app (Logitech Media Server) accessing youtube videos (see here: https://forums.slimdevices.com/showthread.php?105840-ANNOUNCE-YouTube-Plugin-(API-v3)&p=982455#post982455). The media server is accessing youtube via the API (v3) so it has to stay within a quota of 10000 queries per day but since yt-dl is not going via the API, it should not be affected by the quota, right? I did try to give yt-dl its own API (which didn't change anything), probably the limit is not per app but for all my apps together. In any case, when yt-dl doesn't use the API, it should not be affected by the limit, right?

I have no idea how I was able to reach the limit at all and await a response from the developer of the media server youtube plugin on that.

So, I don't know if this is relevant here but I thought I'd mention it.

In any case: since the saily limits for the API are reset at midnight Pacific Time, my I am currently away from reaching that limit but I just tried downloading a video with yt-dl material and it still fails in all three modes: using cookies, using the API and using no authentication at all.

[tophee]

OK, I managed to access the entire logfile: it is currently over 520000 lines long, most of them say this:

ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Could it be that I'm still being haunted by that unfinished subscription? If it has been trying to download stuff from that channel non stop, I wouldn't be surprised if youtube blocked my IP (except that I still can watch youtube videos in the browser). That would also explain why the media server app is having problems accessing yt videos.

How can I stop this from happening?

P.S. Only in the time I wrote this post, the logfile grew by about 2000 lines .... This is crazy!

[Tzahi12345]

The YouTube API key functionality isn't very well documented (or documented at all). I had it in the README in a config explanation section, but it was horribly outdated and so I removed it, but I should make a wiki page where I go through all the settings and give them a proper explanation.

Basically it allows you to search for videos through the URL bar, if you enable it you will see "URL or search" instead of "URL" as the placeholder. If you put in any non-URL text, it does a YouTube search, helping you find videos without a URL.

I wouldn't be surprised if youtube blocked my IP

The 429 errors never actually prevented me from watching YT videos on the web, I'm not sure how they just block non-web viewing. It does sound like you're still blocked by YouTube though. The last thing you can do here is use a proxy with --proxy, or wait the 429 out.

If you still have any subscriptions, adding --download-interval 60 to their args is another good mitigation (reminder that I still need to add the ability to edit subscriptions!). This will download the videos once every minute; it apparently pisses off YT less often. You can actually add it manually if you wish by editing the users.json file if you're in multi-user mode, or db.json if you're not. This might be a big file, so CTRL+F "custom_args" might be helpful.

P.S. Only in the time I wrote this post, the logfile grew by about 2000 lines .... This is crazy!

Ballooning logs is a bit of an issue. I don't know if there's a way to easily merge logs like so without doing it manually:

429 error
429 error
429 error
some other log
429 error
429 error

into

429 error (3)
some other log
429 error (2)

Google Chrome does it pretty well. This will compress the logs considerably, especially for those with repeating errors.

[tophee]

Interesting info about how the youtube api is integrated, thanks.

But there are two issues here that are worth considering separately (which is why I created separate tickets): rendering problems with big log files and the reason why so much is being logged. This ticket was meant to be about the latter, i.e. why does the app keep bombarding yt with requests when it's been told by yt to @#%& off. If yt says "too many requests", does it make sense to keep sending requests?

And then, of course, the question is: how do I manually stop it from doing that. I think I managed now by deleting and recreating the container like you said earlier plus hiding all the config files so that they would be recreated.

I will look at those json files for editing subscriptions. It sounds like that might also be the way to stop a huge download manually? Or is the json file only read once at the beginning of the download and then never again?

[Tzahi12345]

This ticket was meant to be about the latter, i.e. why does the app keep bombarding yt with requests when it's been told by yt to @#%& off. If yt says "too many requests", does it make sense to keep sending requests?

Hm, so you suggest holding off on subscriptions until the 429's go away? (maybe do a quick check once every few hours or so to see if the 429 is still there).

And then, of course, the question is: how do I manually stop it from doing that. I think I managed now by deleting and recreating the container like you said earlier plus hiding all the config files so that they would be recreated.

I will look at those json files for editing subscriptions. It sounds like that might also be the way to stop a huge download manually? Or is the json file only read once at the beginning of the download and then never again?

It's only read at the beginning, unfortunately. But there is a good chance killing the process is possible, it's something I need to test. There was a suggestion at some point to make it obvious a subscription is getting new videos (like a progress bar). If that's added, I can add a simple button that can kill the process manually. Better yet, I can also make it so unsubscribing auto kills the downloading process as well.

Anyways, I'll edit the issue title to better reflect what's needed. Hopefully we can get this in sooner rather than later.

[tophee]

I'm glad I was able to contribute with some usefull suggestions. In terms of urgency, I think the most important thing is to have downloads killed when unsubscribing. Then comes the "Stop downloads" button for any kind of download.

Although I would really like to have an edit option for subscriptions, I think that can wait a bit, given that you already can edit the config files. But maybe mention that somewhere?

Hm, so you suggest holding off on subscriptions until the 429's go away? (maybe do a quick check once every few hours or so to see if the 429 is still there).

Yes, that would make sense to me. But then again I have no idea how exactly triggers the ban that is causing the 429 or whether continuing requests during the ban will prolong it, but even if sending many requests doesn't do anything bad, it certainly doesn't any good either (except, maybe, for resuming the download a few minutes earlier, but even that is assuming that such a ban is actually a matter of minutes. In my case, it seems to be more like a matter of days (My downloads are still failing).

[Tzahi12345]

I'm glad I was able to contribute with some usefull suggestions. In terms of urgency, I think the most important thing is to have downloads killed when unsubscribing. Then comes the "Stop downloads" button for any kind of download.

I've looked more into adding the ability to stop a specific download -- unfortunately it isn't very easy to do with the current youtube-dl library. Interestingly enough, for the "unsafe" download, it's possible to stop midway, but for the "safe" downloads it simply isn't (the safe method calls the binary directly). I've asked about adding this ability here on the node-youtube-dl GitHub.

What we can do is get all the youtube-dl processes currently running and kill those, which is a bit crude but it gets the job done. You can see the commit here, I basically added a button in the Downloader section in the settings to kill all downloads, check it out in the latest nightly! In the future, I'll expand this to filter by args and just kill a sub's downloads when unsubscribing.

But then again I have no idea how exactly triggers the ban that is causing the 429 or whether continuing requests during the ban will prolong it

That's my fear. I really don't know how else to address this error, other than providing the resources/research that I've done over the last 6 months or so. Unfortunately it isn't something we can control, but maybe adding the --sleep-interval SECONDS arg by default for subscriptions is necessary (let users remove it if they want). That, and providing proper error messages (429 doesn't give much info) is the best we can do for now.

[jonathanm-ur]

Adding my 2 cents here, I'm having the same issues, where subs are triggering repeated 429's after a couple days of working fine. Is there any way you could detect that type of error, and possibly put a "Paused for reasons" notification on the main page, along with a "Resume" button?

I can't get a handle on how long it takes to time out and start working again, but I've been starting the program, and getting immediate 429's and shutting back down for almost 5 days now. I'm wondering if the number of denied requests has any bearing on the wait time, which is why I'd like the program to pause immediately when it detects a sequence of repeated 429's instead of continuing to hammer the servers.

[GlassedSilver]

YouTube throttling is notoriously poorly documented...

...for the very reason that YT does not want you to basically know an exact wait time and then hit the servers again with full force. Well, that's my educated guesstimating at least.

It also depends on how the video is accessed, I've read on yt-dlp's issue tracker that sometimes you may solve a specific download being too slow by opening the video in your browser, scrubbing to the position your download is at and then playing at the same time to I guess you could say "validate" your IP's request by coming from a known browser's origin.

I also read that usually it's not even the amount of downloaded "mass", but the amount of calls generated by reading the info for downloads. At least that's what I read over at yt-dlp's issue tracker.

yt-dlp/yt-dlp#985 (comment) also has a bit more info on this.

Further problematic cases

Curiously, what can even add fuel to the fire of this issue is to pause the container/app (if you run bare-metal) and restart after a few days.

Then your subs kicking in all at once with more to download since your backlog grew even more may increase the issue.

Possible fix for this

What I'd do is:

1. pause some subscriptions, maybe even all of them if needed,
2. then pausing the app and
3. if you have a dynamic IP changing it or waiting a few days.
   (I've also heard of people getting throttled by YT for WEEKS, but I think that's an extreme case and I doubt that anyone runs an instance that generates so much traffic for YT that they would ban your IP for that long)
4. Run the app again, turning on subs one by one. This should reduce the load that is caused by the backlog of stuff to download by thinning it out.

!! Absolutely also reconsider your check interval for subscriptions. !!

About what YTDL-M can do

• I definitely think that the application should register throttle errors and offer to dynamically pause and proceed with expanded check intervals so unmonitored use cases stall less and uptime of scraping is handled more efficiently, since pausing the app for days can, as I laid out, be counterproductive if not done right and IMHO sub aficionados will definitely agree that better error 429 handling would go a LONG way to not only making YTDL-M more reliable, but also getting a USP (unique selling point), because I don't know of any other project that actively smartly throttles on 429s for the users.

• Of course occurrences of throttle error codes should also be visualized better as error to - if desired - manually act on. At least I think it could add some valuable feedback to the user and cause less issues about throttling that we could instead actively assist the user with by showing possible fixes in-app. :)

[jonathanm-ur]

I'll just add a quick note, if you modify how this is handled, please allow us to detect and handle it externally, hopefully I can continue to use my current script which checks the last few lines of logs for 429 errors, if it finds them, it stops the container, restarts the VPN to get a new address, then restarts the container. I have ZERO issues with 429's using this method. After I implemented my script, I went from not being able to consistently maintain a handful of subs to being able to have dozens checked very regularly.

What I guess I'm asking for is the ability to leave the 429 as being the last bit in the log so I can kill and restart the container based on that, as well as whatever you do in the app to mitigate the issue for those who can't easily and automatically change IP's.

[GlassedSilver]

That's why I said we should think about offering to handle this automatically. :)

In any case, even with internal logic I think we could register a change of IP to reset any throttling, since unless you use cookies or the API you do have a "new identity" with a new IP, however it was obtained, and probably remedies the need to throttle on our end. :)