[Youtube Issue] Error when loading from YouTube (429/captcha form)

[jagrosh]

Describe the bug
YouTube has recently starting enforcing ratelimits. This is likely because streaming the data directly harms their service (as they receive no ad revenue when videos are streamed in this way). When your IP is ratelimited, no data from YouTube can be loaded, and attempting to load anything from YouTube will result in an error.

Does this affect you?

• If you see Invalid status code for video page response: 429 in error messages, it means that your IP has been ratelimited.
• If you see Sorry for the interruption. We have been receiving a large volume of requests from your network. in error messages, it means that YouTube is is requesting a captcha for your requests.

Solution
There is no solid solution to this at the moment. Your best options are:

• Turn off the bot for a few days (and don't attempt to use anything that connects to YouTube from the same IP address)
• Host from a different IP address

Version Info
This affects all connections to YouTube, regardless of which version of JMusicBot is used.

Additional Info
Here's the information that YouTube/Google provides when a music bot (or any service really) gets ratelimited:

Our systems have detected unusual traffic from your computer network. Please try your request again later. Why did this happen?
This page appears when Google automatically detects requests coming from your computer network which appear to be in violation of the Terms of Service. The block will expire shortly after those requests stop.
This traffic may have been sent by malicious software, a browser plug-in, or a script that sends automated requests. If you share your network connection, ask your administrator for help — a different computer using the same IP address may be responsible. Learn more
Sometimes you may see this page if you are using advanced terms that robots are known to use, or sending requests very quickly.

[TheKleopiNetwork]

Add the option for a local persistant music cache so music from youtube has to only be streamed (and saved) once.
Should reduce the load of the autoplaylist and frequently requested tracks.
Mind licenses.

[jagrosh]

A cache of songs is something I've thought about before (and will continue to think about), but there are several issues.

Lavaplayer (the internal library used for loading youtube and other sources) is strictly a streaming tool, and thus the songs that are streamed never have associated files. To cache music, all of the streamed packets would need to be collected, encoded, and saved to a valid format. This would technically be possible, but would break down when streaming live sources or when seeking within tracks. Therefore, the songs would need to be downloaded separate from the stream, which would mean potentially worsening the ratelimit issue (depending on when things were downloaded).

This would additionally require a lot of work to write such a downloading & encoding system, or adding new dependencies like python & youtube-dl to handle the downloading. This would also require a new searching mechanism to handle caching the results of searches (as even searching - not even playing - can affect ratelimiting).

Other minor issues include disk space and search results changing over time, both of which could potentially be solved by having a time- or disk-size-based expiry system for the cache. Licenses and copyrighted content would be another issue.

Overall, I think the easiest solution for the majority of people would be to just play more local files; use local files or mp3 urls in playlists, and keep other local files under simple and easy-to-access filenames. Adding play shortcuts (key-value pairs of keyword to song uri) might be a feature to assist with this as well.

[johncsuti]

This is true, I have been hit with a rate limit but my friend that works for google’s YouTube department said that if you fake a browser agent it will make it look like a normal person. This will not initiate a rate limit as soon or at all in some cases. Maybe something we should look into. I have connected my system to a squid proxy that fakes the browser agent and have not been hit since. The bot has been running on YouTube every day since May.

[johncsuti]

This is also promising
sedmelluq/lavaplayer#232

[jagrosh]

if you fake a browser agent it will make it look like a normal person

I believe lavaplayer already does this, but this is definitely not the only factor. Google has different limits for different IP ranges as well. For example, running the bot on a home network (consumer IP block) will be far less-likely to get locked out than running it from a known cloud network.

This is also promising
sedmelluq/lavaplayer#232

That doesn't look promising for small, self-hosted bots. It seems more aimed towards larger bots that likely have a lot of IP addresses at their disposal (or can afford to purchase blocks of IPs) that can be cycled through. I don't IP cycling will be useful for the average JMusicBot user, who likely hosts on their home network or a VPS with only one IP (or even a shared IP).

Also, it seems that another new issue is popping up: sedmelluq/lavaplayer#248

Overall, it seems like Google is very intent on stopping music bots & other similar applications, so my instinct would be to move away from using YouTube as the primary source for songs instead of constantly jumping through ever-changing hoops.

[streamkiller]

https://hastebin.com/oguyocumet.xml

As requested console ouput of the error

[streamkiller]

I've copied the complete folder and ran it on another pc and it worked like a charm, I am gonna look into it if I can find the issue that causes this.

I will do a system restore to 11 days ago when it was working good, the programs that were updated are these (top list is currently installed bottom how it was 11 days ago):
https://gyazo.com/33f94c18e1cc4876380c4280477d34d9

[jagrosh]

Based on the console output, this looks like that is a similar issue; same cause (a lot of traffic) but instead of a 429, YouTube is requesting you complete a captcha. You might be able to complete that in your browser, but I don't think there is any solid/long-term solution other than just keeping the bot offline for a while, and making sure that you're not opening too many streams at once.

<p class='largeText'>Sorry for the interruption. We have been receiving a large volume of requests from your network.</p>

<p>To continue with your YouTube experience, please fill out the form below.</p>

[johncsuti]

Yeah the captcha would only complete for one request to go through not anymore so even if he completed it it would not help

[Technetium1]

What if we were willing to pay a captcha solving service? Any possibility of that?

[jagrosh]

I think that's a bit out of the scope of a personal music bot. If anything, I think the solution to captchas would be to somehow forward them to the bot owner or the person trying to play the song, but even that could be difficult. Also, that would only solve a subset of the issues that people are facing with YouTube (it wouldn't solve ratelimits or other video blockages).

[johncsuti]

Just download your favorite songs off YouTube and play them using Music-bot from a wav file.

[Tavinen]

I use windows 10 64 bit and i run it on my normal computer

[jagrosh]

That is not related to this issue, that just means you aren't using 64-bit java

[Ad-Blokker]

I think this ties into it also:
sedmelluq/lavaplayer#460

^ Fixed with dependencies update of the latest release

[johnb-xp]

I downloaded my whole default playlist using YouTube-dl, working great. Super easy to do with https://github.com/MrS0m30n3/youtube-dl-gui because I'm not too experienced with terminal programs.

[RickeyJLancaster]

I just installed the bot and got rate-limited when loading a playlist :s
Wouldn't be better to load only the present song and the next one to prevent that? Should I open a separated issue?

[onlyyouX2]

I run the bot but sometimes it works and some do not , and I noticed something when searching for something in the English, Arabic language that does not work and there’s a mistake occurs, I do not think that the connection ban because I tried it on my device and private servers , and the same problem I was trying all versions of the bot so I had no problems except After the new updates

[onlyyouX2]

Mr.jagrosh

You can make sure at first it will start, then wait for a few mistakes, but you must leave the bot in the room

[onlyyouX2]

I contact with new ip , but i has the same problem and idk the reason . I’m sure there’s something must be fixed in bot . Also i chatted my friends and there have the problem too .

Should I open New issue ?

[17:15:17] [INFO] [Startup]: Loaded config from C:\Users\Administrator\Desktop\Bot\config.txt
[17:15:18] [INFO] [JDA]: Login Successful!
[17:15:19] [INFO] [WebSocketClient]: Connected to WebSocket
[17:15:20] [INFO] [JDA]: Finished Loading!
[17:16:23] [ERROR] [LocalAudioTrackExecutor]: Error in playback of M6z0Qql4-qo
com.sedmelluq.discord.lavaplayer.tools.FriendlyException: Something broke when playing the track.
at com.sedmelluq.discord.lavaplayer.tools.ExceptionTools.wrapUnfriendlyExceptions(ExceptionTools.java:43)
at com.sedmelluq.discord.lavaplayer.track.playback.LocalAudioTrackExecutor.execute(LocalAudioTrackExecutor.java:117)
at com.sedmelluq.discord.lavaplayer.player.DefaultAudioPlayerManager.lambda$executeTrack$1(DefaultAudioPlayerManager.java:349)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.RuntimeException: java.io.IOException: Invalid status code for video page response: 429
at com.sedmelluq.discord.lavaplayer.source.youtube.DefaultYoutubeTrackDetailsLoader.loadDetails(DefaultYoutubeTrackDetailsLoader.java:28)
at com.sedmelluq.discord.lavaplayer.source.youtube.YoutubeAudioTrack.loadBestFormatWithUrl(YoutubeAudioTrack.java:75)
at com.sedmelluq.discord.lavaplayer.source.youtube.YoutubeAudioTrack.process(YoutubeAudioTrack.java:42)
at com.sedmelluq.discord.lavaplayer.track.playback.LocalAudioTrackExecutor.execute(LocalAudioTrackExecutor.java:105)
... 4 common frames omitted
Caused by: java.io.IOException: Invalid status code for video page response: 429
at com.sedmelluq.discord.lavaplayer.source.youtube.DefaultYoutubeTrackDetailsLoader.load(DefaultYoutubeTrackDetailsLoader.java:39)
at com.sedmelluq.discord.lavaplayer.source.youtube.DefaultYoutubeTrackDetailsLoader.loadDetails(DefaultYoutubeTrackDetailsLoader.java:26)
... 7 common frames omitted
[17:36:03] [ERROR] [LocalAudioTrackExecutor]: Error in playback of py8bXg1DUEE
com.sedmelluq.discord.lavaplayer.tools.FriendlyException: Something broke when playing the track.
at com.sedmelluq.discord.lavaplayer.tools.ExceptionTools.wrapUnfriendlyExceptions(ExceptionTools.java:43)
at com.sedmelluq.discord.lavaplayer.track.playback.LocalAudioTrackExecutor.execute(LocalAudioTrackExecutor.java:117)
at com.sedmelluq.discord.lavaplayer.player.DefaultAudioPlayerManager.lambda$executeTrack$1(DefaultAudioPlayerManager.java:349)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.RuntimeException: java.io.IOException: Invalid status code for video page response: 429
at com.sedmelluq.discord.lavaplayer.source.youtube.DefaultYoutubeTrackDetailsLoader.loadDetails(DefaultYoutubeTrackDetailsLoader.java:28)
at com.sedmelluq.discord.lavaplayer.source.youtube.YoutubeAudioTrack.loadBestFormatWithUrl(YoutubeAudioTrack.java:75)
at com.sedmelluq.discord.lavaplayer.source.youtube.YoutubeAudioTrack.process(YoutubeAudioTrack.java:42)
at com.sedmelluq.discord.lavaplayer.track.playback.LocalAudioTrackExecutor.execute(LocalAudioTrackExecutor.java:105)
... 4 common frames omitted
Caused by: java.io.IOException: Invalid status code for video page response: 429
at com.sedmelluq.discord.lavaplayer.source.youtube.DefaultYoutubeTrackDetailsLoader.load(DefaultYoutubeTrackDetailsLoader.java:39)
at com.sedmelluq.discord.lavaplayer.source.youtube.DefaultYoutubeTrackDetailsLoader.loadDetails(DefaultYoutubeTrackDetailsLoader.java:26)
... 7 common frames omitted

[jagrosh]

If you get a 429 (as per your logs), it is this issue, there's no need to open a duplicate

[KenaiTheWolf]

So, I've ran into this issue as well as of late. I've found that the auto playlist command tends to aggravate YouTube into striking the bot down. Currently, my personal work around for this was to fork this project and create a branch of my fork which has the command disabled (it's a pretty janky system on my end, and I haven't tested it yet.)

I made it such that the event sends an error message instead and explains that I've disabled the feature. Going to test tomorrow to make sure I didn't break anything.

[DemonFiend]

Just got this error after like 1-2 years of using the bot. Curious if there's a work around?

Seems to have happened to me once i added 200 YT Videos to the playlist was at 130 before.

If not I was wondering if there is some way we could link this through a VPN in like an advanced setting. (Ideally a config option to use 'default ip' or 'custom ip=XXX.XXX.XXX.XXX') to redirect the bot through a VPN would be ideal).

If not i can probably setup a VMBox and try and redirect the traffic through the vpn that way, but that maybe more complex & im not sure how that would affect the bot.

[ribbanya]

Is there a reason the bot can't cache http requests? I set this up in python the other day and it was extremely easy.

Java solution (I haven't looked into it extensively)

Obviously this would get pretty big, so you can make it optional, set a size limit, and allow configuring expiration periods, but it would help heavy bot users at least.

[adamhradil]

What is the current progress on this? I'm getting some warning messages, but the bot generally functions fine.

[21:39:27] [WARN] [RateLimiter]: Encountered 429 on route PATCH/channels/{channel_id} with bucket xyz:channel_id=xyz Retry-After: 262000 ms
[19:57:58] [WARN] [RateLimiter]: Encountered 429 on route PATCH/channels/{channel_id} with bucket xyz:channel_id=xyz Retry-After: 467000 ms
[20:56:00] [WARN] [RateLimiter]: Encountered 429 on route PATCH/channels/{channel_id} with bucket xyz:channel_id=xyz Retry-After: 108000 ms
[21:11:56] [WARN] [RateLimiter]: Encountered 429 on route PATCH/channels/{channel_id} with bucket xyz:channel_id=xyz Retry-After: 369000 ms
[21:26:44] [WARN] [RateLimiter]: Encountered 429 on route PATCH/channels/{channel_id} with bucket xyz:channel_id=xyz Retry-After: 240000 ms
[22:27:29] [WARN] [RateLimiter]: Encountered 429 on route PATCH/channels/{channel_id} with bucket xyz:channel_id=xyz Retry-After: 482000 ms
[23:22:28] [WARN] [RateLimiter]: Encountered 429 on route PATCH/channels/{channel_id} with bucket xyz:channel_id=xyz Retry-After: 288000 ms
[23:50:29] [WARN] [RateLimiter]: Encountered 429 on route PATCH/channels/{channel_id} with bucket xyz:channel_id=xyz Retry-After: 586000 ms
[00:11:03] [WARN] [RateLimiter]: Encountered 429 on route PATCH/channels/{channel_id} with bucket xyz:channel_id=xyz Retry-After: 233000 ms
[00:31:02] [WARN] [RateLimiter]: Encountered 429 on route PATCH/channels/{channel_id} with bucket xyz:channel_id=xyz Retry-After: 404000 ms
[00:54:24] [WARN] [RateLimiter]: Encountered 429 on route PATCH/channels/{channel_id} with bucket xyz:channel_id=xyz Retry-After: 382000 ms
[01:10:26] [WARN] [RateLimiter]: Encountered 429 on route PATCH/channels/{channel_id} with bucket xyz:channel_id=xyz Retry-After: 293000 ms

[jagrosh]

@adamhradil those logs are not related to this issue; those logs are the result of the bot attempting to put the "now playing" info in a channel topic (which itself is the result of the bot having the Manage Channels permission in the channel in which you've used settc).