Steam using SSL for downloads, no caching

[C9Glax]

Describe the issue you are having

I am using the Linux-client for Steam and lancache didn't seem to work suddenly (after switching from windows).
Most of the connections are made via port 443 and therefor not cached. Only after the third or fourth re-run I have now gotten some connections via port 80 (but still only around half of speeds I have historically gotten).

How are you running the container(s)

I run monolithic with a pihole dns to point to monolithic.

DNS Configuration

The DNS-server is pihole and resolves lancache.steamcontent.com properly to the correct address (verified with dig).

;; ANSWER SECTION:
lancache.steamcontent.com. 0    IN      A       192.168.0.5

Since uklans/cache-domains got a recent PR which removes most of the addresses, I thought this might be the problem, and added the old addresses back. Which did not only partially seem to resolve the problem.

Output when starting a download

192.168.0.5 = lancache.steamcontent.com
192.168.0.50 = PC running Steam

On PC

netstat -natp | grep steam

tcp        0      0 127.0.0.1:57343         0.0.0.0:*               LISTEN      2976/steam
tcp        0      0 127.0.0.1:27060         0.0.0.0:*               LISTEN      2976/steam
tcp        0      0 192.168.0.50:42085      192.168.0.5:443         ESTABLISHED 2976/steam
tcp        0      0 192.168.0.50:54135      192.168.0.5:443         ESTABLISHED 2976/steam
tcp        0      0 192.168.0.50:56793      192.168.0.5:443         ESTABLISHED 2976/steam
tcp        0      0 192.168.0.50:40845      192.168.0.5:80          ESTABLISHED 2976/steam
tcp        0      0 192.168.0.50:42301      192.168.0.5:443         ESTABLISHED 2976/steam
tcp        0      0 192.168.0.50:56269      192.168.0.5:443         ESTABLISHED 2976/steam
tcp        0      0 192.168.0.50:53229      155.133.226.78:27029    ESTABLISHED 2976/steam
tcp        0      0 192.168.0.50:34029      192.168.0.5:80          ESTABLISHED 2976/steam
tcp        0      0 192.168.0.50:51291      192.168.0.5:443         ESTABLISHED 2976/steam
tcp        0      0 192.168.0.50:41629      192.168.0.5:80          ESTABLISHED 2976/steam
tcp        0      0 127.0.0.1:27060         127.0.0.1:39374         ESTABLISHED 2976/steam
tcp        0      0 192.168.0.50:48057      192.168.0.5:80          ESTABLISHED 2976/steam
tcp        0      0 192.168.0.50:41615      192.168.0.5:80          ESTABLISHED 2976/steam
tcp        0      0 192.168.0.50:42167      192.168.0.5:443         ESTABLISHED 2976/steam
tcp        0      0 192.168.0.50:48281      192.168.0.5:443         ESTABLISHED 2976/steam
tcp        0      0 192.168.0.50:50247      192.168.0.5:80          ESTABLISHED 2976/steam
tcp        0      0 192.168.0.50:54811      192.168.0.5:80          ESTABLISHED 2976/steam
tcp        0      0 192.168.0.50:44659      192.168.0.5:80          ESTABLISHED 2976/steam

On Lancache-server

netstat -natp | grep 192.168.0.50

tcp        0     84 192.168.0.5:80          192.168.0.50:44765      ESTABLISHED -
tcp        0      0 192.168.0.5:80          192.168.0.50:42009      ESTABLISHED -
tcp        0      0 192.168.0.5:80          192.168.0.50:34029      TIME_WAIT   -
tcp        0      0 192.168.0.5:443         192.168.0.50:54135      ESTABLISHED -
tcp        0      0 192.168.0.5:80          192.168.0.50:40845      TIME_WAIT   -
tcp        0      0 192.168.0.5:443         192.168.0.50:56269      ESTABLISHED -
tcp        0      0 192.168.0.5:443         192.168.0.50:42085      ESTABLISHED -
tcp        0    460 192.168.0.5:80          192.168.0.50:40375      ESTABLISHED -
tcp        0    660 192.168.0.5:80          192.168.0.50:54811      ESTABLISHED -
tcp        0      0 192.168.0.5:443         192.168.0.50:46529      ESTABLISHED -
tcp        0      0 192.168.0.5:80          192.168.0.50:44659      ESTABLISHED -
tcp        0      0 192.168.0.5:443         192.168.0.50:42167      ESTABLISHED -
tcp        0      0 192.168.0.5:443         192.168.0.50:42301      ESTABLISHED -
tcp        0    412 192.168.0.5:80          192.168.0.50:41615      ESTABLISHED -
tcp        0    252 192.168.0.5:80          192.168.0.50:48057      ESTABLISHED -
tcp        0      0 192.168.0.5:443         192.168.0.50:48281      ESTABLISHED -
tcp        0      0 192.168.0.5:443         192.168.0.50:56793      ESTABLISHED -
tcp        0      0 192.168.0.5:80          192.168.0.50:50247      ESTABLISHED -
tcp        0      0 192.168.0.5:22          192.168.0.50:39192      ESTABLISHED -
tcp        0      0 192.168.0.5:22          192.168.0.50:34892      ESTABLISHED -

[C9Glax]

I believe this is not an issue with lancache per-se, but you would have better connections at Valve to look into why they are trying to transfer game-files via https.

[C9Glax]

What seems to have helped is actually reverting the mentioned PR, thus forcing http from lancache.

[mintopia]

I've addressed this in the issue on cache-domains, but essentially:

Reverting that PR and using the old cache domains won't help with caching. If Steam detects an error using a lancache - a 404 or other error, possibly from the upstream CDN, it will switch back to going directly to the content servers for a few minutes, bypassing the lancache, and may upgrade the connection back to HTTPS.

If this happens - even if you had the full list of hostnames, all that is happening is the HTTPS traffic is now being passed through your LAN Cache and will not be cached.

To see if this is the problem here, check the content_log.txt file in your Steam directory. If you have lines saying Forgetting cache type 'LANCache'... with a reason, then it means Steam has stopped using the LanCache and is going direct to the CDN and maybe using HTTPS.

We have no control over this behaviour of the Steam client and it is intended behaviour by Valve.

Check the logfile and let me know if you are seeing messages like that in it.

[1nd1r4]

I also have the problem, that Steam randomly switches to SSL downloads. Sometimes this is due to the cache "hanging" for a few seconds when downloading a game for the first time but usually its completely out of the blue. I have no explanation for this behaviour.

===CACHE===

==> /mnt/hdd/lancache/logs/error.log <==
2022/12/27 17:05:15 [error] 1898#1898: *720265 upstream timed out (110: Connecti on timed out) while reading response header from upstream, client: 10.1.3.46, se rver: , request: "GET /depot/292031/chunk/f9ec3884ebf4a5cb4414f4e8235c9d16c05911 cb HTTP/1.1", subrequest: "/depot/292031/chunk/f9ec3884ebf4a5cb4414f4e8235c9d16c 05911cb", upstream: "http://127.0.0.1:3128/depot/292031/chunk/f9ec3884ebf4a5cb44 14f4e8235c9d16c05911cb", host: "steampipe.akamaized.net"
2022/12/27 17:05:15 [error] 1898#1898: *720265 unexpected status code 504 in sli ce response while sending to client, client: 10.1.3.46, server: , request: "GET /depot/292031/chunk/f9ec3884ebf4a5cb4414f4e8235c9d16c05911cb HTTP/1.1", subreque st: "/depot/292031/chunk/f9ec3884ebf4a5cb4414f4e8235c9d16c05911cb", upstream: "h ttp://127.0.0.1:3128/depot/292031/chunk/f9ec3884ebf4a5cb4414f4e8235c9d16c05911cb ", host: "steampipe.akamaized.net"
2022/12/27 17:19:43 [error] 1820#1820: *728911 unexpected status code 426 in slice response while reading response header from upstream, client: 10.1.3.46, server: , request: "GET /depot/292031/chunk/f9ec3884ebf4a5cb4414f4e8235c9d16c05911cb HTTP/1.1", subrequest: "/depot/292031/chunk/f9ec3884ebf4a5cb4414f4e8235c9d16c05911cb", upstream: "http://127.0.0.1:3128/depot/292031/chunk/f9ec3884ebf4a5cb4414f4e8235c9d16c05911cb", host: "google2.cdn.steampipe.steamcontent.com"

==> /mnt/hdd/lancache/logs/access.log <==
[steam] 10.1.3.46 / - - - [27/Dec/2022:17:05:15 +0100] "GET /depot/292031/chunk/ f9ec3884ebf4a5cb4414f4e8235c9d16c05911cb HTTP/1.1" 200 621313 "-" "Valve/Steam H TTP Client 1.0" "MISS" "steampipe.akamaized.net" "-"

==> /mnt/hdd/lancache/logs/upstream-error.log <==
2022/12/27 17:05:15 [error] 1898#1898: *720606 upstream timed out (110: Connecti on timed out) while connecting to upstream, client: 127.0.0.1, server: , request : "GET /depot/292031/chunk/f9ec3884ebf4a5cb4414f4e8235c9d16c05911cb HTTP/1.0", u pstream: "http://2.21.228.155:80/depot/292031/chunk/f9ec3884ebf4a5cb4414f4e8235c 9d16c05911cb", host: "steampipe.akamaized.net"

===CLIENT===

==> C:\Program Files (x86)\Steam\logs\content_log" <==
[2022-12-27 17:05:15] Forgetting cache type 'LANCache' on host '::ffff:a01:10b' due to chunk timeout.
[2022-12-27 17:05:15] IPv6 Connectivity state has changed, downgrading to ipv4-only
[2022-12-27 17:05:15] HTTP (CDN,11) - steampipe.akamaized.net (10.1.1.11:80 / 10.1.1.11:80, host: steampipe.akamaized.net): Received 0 (Invalid) HTTP response for depot 292031
[2022-12-27 17:05:15] HTTP (CDN,11) - steampipe.akamaized.net (10.1.1.11:80 / 10.1.1.11:80, host: steampipe.akamaized.net): Disabling dual-stack for this connection

[C9Glax]

This reply already mentioned the cause.

If Steam detects an error using a lancache - a 404 or other error, possibly from the upstream CDN, it will switch back to going directly to the content servers for a few minutes, bypassing the lancache, and may upgrade the connection back to HTTPS.

However you can circumvent this by actually re-adding all the CDN-Servers, and thus forcing a connection to your LAN-Cache. Which you can do by using your own repo:
CACHE_DOMAINS_REPO
The github repo url for the cache_domains files to use to populate the cdn maps, if you do not want to use the recommended source for testing purposes.
Default: https://github.com/uklans/cache-domains.git
For example I am using my own (reverted) repo: https://github.com/C9Glax/cache-domains

Everything is working as intended by the software and developer, just the user is confused.
So actually: Closing.

[mintopia]

We have spoken to Valve and they have fixed this in the latest update to Steam.