[COMCAST] Images and videos in steam not loading

[jelgersma]

Issue Description:
When in steam store or library it won't load the images and/or movies. I traced it back to being an issue with HTTPS and HTTP.

Recently (5 days ago before this post) an URL got added to uklans/cache-domains repo. Which is the URL "steamcdn-a.akamaihd.net". This URL is the resource URL for videos and images of steam. The URL gets fetched by the DNS server and forwarded to the steamcache docker container (which only listens and runs on port 80) which causes a Connection Refused error.

How to reproduce:
Start latest version of steamcache container and the steamcache-dns container.

Docker commands:
docker run -v /srv/data:/data --name steamcache -d --restart always -p 80:80 steamcache/steamcache

docker run -d --restart always --name steamcache-dns -p ${DNS_IP}:53:53/udp -e USE_GENERIC_CACHE=true -e LANCACHE_IP=${CACHE_IP} steamcache/steamcache-dns:latest

Edit
Current solution that I did was forking steamcache/steamcache-dns and uklans/cache-domains and removing that URL from steam.txt. Build the steamcache/steamcache-dns with docker and ran it the same way you run it usually. Now the images and videos are working again and the caching of game fragments is still working.

[JasonRivers]

Please run the SNIProxy image on port 443 on the same host that is running the steamcache.

You have also defined USE_GENERIC_CACHE but are only running steamcache. This will cause you problems with all other services that are supported from uklans, the GENERIC options will use a single IP for all services and should be used with a reverse proxy or load balancer. To solve this use STEAMCACHE_IP=${CACHE_IP} and remove the USE_GENERIC_CACHE and LANCACHE_IP

[ala53]

Using SNIProxy causes other issues with a Comcast internet connection. Comcast hosts a steam download server at edge.steam-dns-2.top.comcast.net with optional HTTPs.

When SNIProxy is disabled for steam domains, downloads are cached via steamcache but images do not load on steam due to steamcdn-a.akamaihd.net being redirected. When SNIProxy is enabled, images/videos are displayed correctly, however downloads are also proxied through the aforementioned Comcast domain (over HTTP), which means they cannot be cached.

I'm unfamiliar with the steamcache-dns architecture, however I see 3 options to fix this issue

• Blackhole edge.steam-dns{-2}.top.comcast.net in steamcache DNS (which will require changing the bootstrap code)
• Blackhole edge.steamdns... in SNIProxy (I don't think that is possible, however)
• Remove the akamaihd.net domain from the cache-domains list

I think it might be best to remove the akamaihd.net domain as it seems to just be used for images and videos rather than downloads.

[JasonRivers]

We have added the comcast domains to the steamcache-dns image. the latest version of the DNS image will automatically pull in the latest domains from https://github.com/uklans/cache-domains Please update your DNS image. We will not be removing akamaihd.net from the steamcache / dns containers.

[VibroAxe]

@ala53 can you clarify what you mean byt "When SNIProxy is enabled, images/videos are displayed correctly, however downloads are also proxied through the aforementioned Comcast domain (over HTTP), which means they cannot be cached"

With sniproxy enable and the updated dns image you should find that edge.steam-dns-2.top.comcast.net is cached and https traffic is proxied correctly

[ala53]

I'm sorry, I made a small typo -- the quote you mentioned was supposed to read "(over HTTPs)" -- the edge.steam-dns-2.top.comcast.net domain seems to use HTTPs when available (i.e. when SNIProxy is enabled) and is the preferred download server for any Comcast users downloading from steam. Thus, you are stuck with one of two options with steamcache on Comcast internet right now:

• Disable SNIProxy. Downloads are cached, images and videos do not load.
• Enable SNIProxy. Downloads are not cached as they are over HTTPS, images and videos do load.

[ala53]

Quick addendum: I'm running the current version for all the docker images.

[JasonRivers]

Thanks for the clarification!

Can I ask how many users you are running this for?

We've had a quick chat about this in our discord, There's not a great deal we can do if ComCast have decided to use HTTPS, However, there is a tool available to tell steam what CDN to use. This requires that you push your traffic through a squid proxy, I personally haven't used it but give it a go: https://github.com/OpenSourceLAN/steam-hijack

I will warn you that it really does come "as-is", and it may be a little complex to setup.

Another thing you could try on your firewall - if we assume that edge.steam-dns-2.top.comcast.net only stores depot data and not the images, etc. then you could try blocking port 443 to this on your firewall to see if it will fall-back to HTTP - I cant guarantee this, but it might be worth a try.

[ala53]

I'm just running it personally. What I ended up doing was setting up a separate DNS forwarder on the network and routing the edge.steam-dns... addresses to 0.0.0.0, which solved the problem for me as steam could no longer download from the Comcast HTTPs cache.

[miketweaver]

* Blackhole edge.steamdns... in SNIProxy (I don't think that is possible, however)

I've been able to do this by tweaking the SNIProxy config file inside the container.

table {
	edge.steam-dns.top.comcast.net 127.0.0.1:443
	edge.steam-dns-2.top.comcast.net 127.0.0.1:443
	.* *:443
}

I don't know what effect this has on speed. For all I know, the time for the https download to time out could greatly hinder download speeds.

It might also be worthwhile to move the non-depot domains out of the main steam container and into the USE_GENERIC_CACHE container.

---

Another strange solution I came up with:
I gave the Steam Docker 2 IPs. Forwarding both IPs to the container on port 80. Then I exposed SNI only on 1 of the IPs.

Then I edited steamcache-dns to point the comcast addresses to the 2nd IP that did not have SNI in front of it. That solved the issue as well.

It would be nice if the comcast IPs acted like a different Service. It would make it so I don't have to butcher the docker container after every update. Something like COMCASTSTEAM_CACHE would be nice.

[VibroAxe]

@JasonRivers I've been thinking about this and #48 and wondering if we can update the json file to support a blackhole functionality. Even we didn't want to officially support comcast domains, people could at least then perform some of this functionality through a cache-domains fork?

The new RPZ mechanic would also enable this really easily with the rpz cname . which returns NXDOMAIN allowing the client to continue without the domain (rather than any horrible localhost hacks)

[ghost]

hello, I am running into the same issue and see a bunch of great suggestions here but no how to, steps or process to achieve this? I am not a particularly strong linux admin but any help would be greatly appreciated thank you.

[VibroAxe]

@nem1611 Mike has a working proof of concept but we don't currently.support this officially #47 (comment)

[ghost]

@miketweaver this is actually working out as a temp solution, thank you for the suggestion! I can finally run sniproxy while caching steam updates over http

[SinisterSpatula]

@miketweaver this is actually working out as a temp solution, thank you for the suggestion! I can finally run sniproxy while caching steam updates over http

Hi, I want to use steamcache/monolithic and I'm on comcast. I'm confused by the removal of the comcast steam domains from the steam.txt of the cache-domains by @nem1611 . (or the proposed patch to remove them, maybe it's not made it to the current release?). Wouldn't that defeat the purpose and cause it not to cache steam at all? If those are removed and I'm on comcast, that would mean my downloads would go to those comcast steam CDN's and not be intercepted by steamcache, right? That would seem to be a step in the wrong direction, no? If I understand all this, that means I need to add those domains back, and modify the SNIProxy config inside it's container as @miketweaver suggests? How would I go about doing that, if I may ask, if I'm using docker (container station) on a qnap nas.

[SinisterSpatula]

Oh I see, Nem's patch was canceled when he tried out mikes suggestion, so I don't need to worry about adding those domains back. But I'm still a little confused about how to do Mike's suggestion.

[ghost]

@SinisterSpatula you are right, removing the comcast cdn would not have been ideal since the isp forces users over https regardless, which is why I cancelled the request...fortunately leveraging the sniproxy you can block the 443 requests and steam will roll back to 80

you will have to edit the conf file inside the container to make the changes that @miketweaver suggested ie

sudo docker exec -ti 9784bc523df4 tail /etc/sniproxy.conf

sudo docker exec -ti 9784bc523df4 vi /etc/sniproxy.conf

also as a note, the containers will need to run for a bit of time after a reboot or startup before a system connects to it in order for steam to failover to http, you can verify the connect through the access.log

[SinisterSpatula]

Thank you very much nem! Still new to docker and that is very helpful and great info.

[ghost]

np, happy to help since I was muddling my way through this issue recently

[astrolox]

It's my understanding that this is now fixed. PR uklans/cache-domains#19