Circumventing geo-restrictions

[narcolepticinsomniac]

I thought we were already circumventing geo-blocking, but we're not.

Blocked in the U.S.:
https://invidio.us/watch?v=xpWu4_hp00M

[omarroth]

It would be good to start building a list of videos that are "Blocked in US, Blocked in DE, etc".
Do you happen to know in which regions the video linked above is allowed?

[narcolepticinsomniac]

Nah, I can't get it to play using any VPN or proxy. Maybe not a great example, as I suppose it's possible that the link is just broken and the geo-restriction aspect isn't really the issue.

[omarroth]

Here's an example of a video that is allowed in DE but not US: https://invidio.us/watch?v=rV4Mrl2hQro
It appears that in /videoplayback URLs, for example medium quality for the above video YouTube now returns 403 when the ip parameter does not match the requesting IP.

Helpful link: https://www.youtube.com/get_video_info?video_id=rV4Mrl2hQro

[narcolepticinsomniac]

Haven't I seen something like bypassip= params for googlevideo URLs?

[omarroth]

You probably saw ratebypass, I don't happen to know what it does.
A few quick tests:

• I misspoke, YouTube only cares about ip= when the IP is for a video that is not in one of the allowedRegions.
• Attempting to access geo-blocked videos through Tor is fine, as long as the exit node is in the correct region.

[narcolepticinsomniac]

No, I knew I've seen it. Went through my clipboard manager and found one. IDK if it actually works, but:

https://r3---sn-ab5szn7s.googlevideo.com/videoplayback?fvip=3&source=youtube&signature=230E3DC2DF22B86A40021B84ACAD8839AB515422.0D14359F1455503A0BE6096B7B8B1EFB4CD16927&ipbits=0&pl=18&id=o-AIfZC-oGvCCpSym9jRjuh5PzQ3mlPwK2nLKOpIQb6UFj&dur=180.419&mime=video%2Fmp4&key=cms1&ip=2604%3Aa880%3A400%3Ad1%3A%3A180%3Ad001&ei=fGFjW4_5I5SBhwaVtInwDg&c=WEB&requiressl=yes&sparams=dur,ei,expire,id,ip,ipbits,ipbypass,itag,lmt,mime,mip,mm,mn,ms,mv,pl,ratebypass,requiressl,source&itag=22&lmt=1532956145274494&expire=1533261276&ratebypass=yes&redirect_counter=1&rm=sn-ab5eel7s&req_id=ef8047609366a3ee&cms_redirect=yes&ipbypass=yes&mip=00.000.000.000&mm=31&mn=sn-ab5szn7s&ms=au&mt=1533239832&mv=m

(towards the end)

[omarroth]

Doesn't appear to do anything. A while back I put some work into decrypting/resigning /videoplayback URLs but never got anywhere. Notice that in the above URL there is akey and signature, which prevents tampering.

[narcolepticinsomniac]

ipbypass=yes seemed like it may be relevant. I was skeptical, but figured it might be worth mentioning.

[omarroth]

Working on this right now, currently can get info on the videos linked above. https://invidio.us/watch?v=xpWu4_hp00M appears to be allowed in Germany, Japan, Russia, Thailand (of the regions I've tested). The last thing that needs to be done is to proxy the files themselves, although obviously that will be more bandwidth intensive.

[dimqua]

I suggest you to just add this information to the video page, so users can use VPN or proxy to watch videos that are allowed in specific countries.

[omarroth]

That's fair enough. I'm already managing downloads for livestreams, and will do the same for #34 and #90, so I thought I might do that here as well. I do plan on adding info to the video page, something like blacklisted countries or whitelisted countries (depending on which is bigger), and that info is already available though the API.

[omarroth]

Just pushed fada57a, which provides blacklisted/whitelisted countries. I'd take a look at the regions Invidious tries to bypass. It's a pretty small list, but it seems to be able to bypass most of the videos that are geo-restricted. If there are other regions that are allowed by a significant amount of geo-restricted videos, I'll consider adding those regions.
Region codes correspond to those in ISO 3166.

[dimqua]

regions that make up a significant amount of traffic

How can you measure that?

[omarroth]

I think what I mean is the regions that allow a user to watch the largest amount of content. So Germany, US, JP, etc. I can't measure it very well, but I think there is a metric where some percentage of videos are exclusive to region x, so it should be added to the list that Invidious tries to bypass.

I also just clarified my position a bit: the regions that are allowed by the largest number of geo-restricted videos should be added.

[Zero3K]

I wonder if its possible to get it to work without having to proxy the video file itself. If it is, then that would be nice.

[omarroth]

Attempting to access a geo-blocked file returns a 403 as well, so it appears the only solution is to proxy that as well. This is also an issue for #90: Google will block access to the file itself in some instances, so this is likely the only solution unfortunately.

[narcolepticinsomniac]

Not sure if you ever successfully implemented a geo-restriction circumvention. I kinda figured you had bc I hadn't run into any problems in quite a while, but this issue's still open, so perhaps not. Just ran into one though, and besides being geo-blocked, the page takes forever just to load an error message.

https://youtu.be/d9sApXSDtnE

[omarroth]

Currently the system in place is to cycle through proxies in specific regions, so most of the videos mentioned work (with dash enabled so videos are also proxied). The reason it's still open is because there's still a lot of room for improvement.. I expect it to be closed alongside #90.

The reason geo-blocked videos can take a long time to load is if Invidious can't find a proxy or if none of the regions Invidious tries return a valid response. I would like to fix that before I close this.

[TheFrenchGhosty]

A 20$ bounty has been added to this issue.

Anyone opening a PR fixing this issue, will receive 20$ (in BTC) from the Invidious project.

More details: #1898

[unixfox]

We could use hola proxies which have a lot of available locations: https://github.com/Snawoot/hola-proxy

[TheFrenchGhosty]

@unixfox Tor is maybe more reliable than random proxy, maybe?

[unixfox]

@unixfox Tor is maybe more reliable than random proxy, maybe?

You can't choose the country exit node on Tor.

[outloudvi]

You can't choose the country exit node on Tor.

I think you can.

[syeopite]

Although Tor is a viable solution, we're going to be consuming a large portion of bandwidth on the already limited Tor network; possibly handicapping connections for those who actually needs Tor.

Not to mention the fact that Tor is incredibly slow. If this is the solution we decide to use then this should definitely be configurable imo.

[unixfox]

You can't choose the country exit node on Tor.

I think you can.

Ok but:

• Tor is slow, and it's a video stream that we are fetching not a simple HTML page. We won't be able to load 720p videos smoothly.
• In how many countries are tor exit nodes hosted? More than 40 countries compared to hola proxy?
• Tor exit nodes are often blocked on Google services.

[SamantazFox]

@unixfox Tor is maybe more reliable than random proxy, maybe?

We shouldn't use Tor. It wasn't designed to handle such a load, and the current network doesn't have the ressources to be used as a video proxy.

See this blog post for more details and the related section in the FAQ

[github-actions]

This issue has been automatically marked as stale and will be closed in 30 days because it has not had recent activity and is much likely outdated. If you think this issue is still relevant and applicable, you just have to post a comment and it will be unmarked.

[unixfox]

One idea that comes to my mind when discussing with other people on matrix is that we could say to the user that this video is available in this other instance because we fetched the countries from the API of the instances list.

[syeopite]

We might have to depend on an external service that categorizes what regions a certain video is allowed in for that considering how YouTube sometimes just doesn't return anything sometimes besides a region-locked error message.

[unixfox]

We might have to depend on an external service that categorizes what regions a certain video is allowed in for that considering how YouTube sometimes just doesn't return anything sometimes besides a region-locked error message.

Piped have a special external service that use demo official key on YouTube official API for checking the regions where the video is available: https://github.com/TeamPiped/region-restriction-checker

I have also thought of using www.google.com, force a country in the settings and search for the video ID in the "videos" tab, then check if the video appears. But it doesn't work well.

And last thoughts, something I have not yet tested, https://www.tunnelbroker.ch/ allows to configure a country when creating a new IPv6 subnet, I wonder if YouTube takes into account this information that is written in the whois or not.

[syeopite]

https://github.com/TeamPiped/region-restriction-checker

This is great! We should find someway of integrating that or something similar we built ourselves into Invidious. I am a bit concerned about calling their official API considering YouTube's recent legal actions but if its an external service and not Invidious directly it should probably be fine.

https://www.tunnelbroker.ch/ allows to configure a country when creating a new IPv6 subnet...

If that works then that's another alternative and one that is definitely safer than using YouTube's official API.