Proton 5.13 can not handle /usr/local/games steam library folder

[rcpoison]

Can not get anything working with proton 5.13 while having a separate "Steam Library Folder"

pressure-vessel-adverb[95287]: Failed to execute child process "/usr/local/games/Steam/steamapps/common/SteamLinuxRuntime_soldier/pressure-vessel/bin/pressure-vessel-launcher" (No such file or directory)
ln: failed to create symbolic link '/tmp/SteamPVSockets.jMXXWs/SteamLinuxRuntime.f2d469346ba3105c/socket' -> '': No such file or directory
...
bwrap: Can't chdir to /usr/local/games/Steam/steamapps/common/Torchlight: No such file or directory

$ ls -lah /usr/local/games/Steam/steamapps/common/SteamLinuxRuntime_soldier/pressure-vessel/bin/pressure-vessel-launcher
-rwxr-xr-x 1 poison users 118K Oct 27 11:59 /usr/local/games/Steam/steamapps/common/SteamLinuxRuntime_soldier/pressure-vessel/bin/pressure-vessel-launcher

$ ls -lahd /usr/local/games /usr/local/games/Steam /usr/local/games/Steam/steamapps/common/Torchlight
drwxr-xr-x 1 poison users 250 Oct 27 12:53 /usr/local/games
drwxr-xr-x 1 poison users  18 Oct 27 12:37 /usr/local/games/Steam
drwxr-xr-x 1 poison users 1.1K Oct 27 11:45 /usr/local/games/Steam/steamapps/common/Torchlight

$ mount|grep games
/dev/mapper/ssdvg0-games on /usr/local/games type btrfs (rw,noatime,compress=zstd:3,ssd,space_cache,subvolid=5,subvol=/)

Steps taken to reproduce:

• delete ~/.steam and ~/.local/share/Steam
• run steam, add Library Folder on different mountpoint, make default
• force Proton-5.13-1 for game and install

[rcpoison]

Uhm, OK, if I mount --bind the directory to either a subdir in my home or somewhere completely different like /data/games it works.
Does bwrap not like it being in /usr/local or something?

[kisak-valve]

Hello @rcpoison, let's treat this as a pressure vessel issue until there's a stronger indication that the issue is elsewhere.

[smcv]

Does bwrap not like it being in /usr/local or something?

Almost! bwrap doesn't care which path we use, but the pressure-vessel container-launcher (which is what runs bwrap) replaces /usr with the runtime that is used to run Proton. This is how the container runtime can make sure Proton gets the correct libraries. /usr is mounted read-only (to avoid accidentally changing the runtime, which would lead to unpredictable bugs that are difficult to solve), and can't have arbitrary things from the host system mounted below it, so the container setup code treats /usr as reserved, and ignores attempts to share filesystems below /usr with the container. The container setup code is borrowed from Flatpak, which has a similar design and the same restriction.

As a result, /usr/local/games/Steam/steamapps doesn't appear inside the container, therefore /usr/local/games/Steam/steamapps/common/Torchlight doesn't exist, and when we ask to change to that directory, it can't work.

Your real /usr does appear inside the container at /run/host/usr, but, again, it's read-only. Also, if we used that path, games and Proton would be likely to save configuration files that contain /run/host/usr, which wouldn't work when not using the container runtime.

/usr/local isn't a particularly great match for a Steam library, because /usr/local is designed to be system-wide, shared between all users, and "usually" accessed in a read-only way (the only writes into /usr/local are when the sysadmin installs or updates locally-installed software as root), whereas Steam libraries are owned by a particular user and are "usually" read/write to the user that owns them.

I'd recommend putting Steam libraries on a FHS system below /home, /srv or /media, or maybe in /opt or a non-FHS top-level location like the /data you used to work around this. All of those paths should be fine for pressure-vessel.

It would probably be possible to make a special-case exception for /usr/local, but the more special cases we pile into pressure-vessel, the more complex and less robust it gets. So I'd prefer to spend our "complexity budget" on things that are harder to work around, like making it work on more operating systems and under Flatpak.

Making pressure-vessel fail with a clearer warning/error when asked to share a path below /usr would probably be a good thing to have, though.

[rcpoison]

@smcv thanks for the detailed explanation.

/usr/local/games is just an artifact since my system wasn't reinstalled for 13 years with old games that installed themselves there and I wanted all games on a dedicated disk - this was the lazy solution :)

Bind mount works fine. Thanks again.

[LubosD]

@smcv I also have my SteamApps directory moved elsewhere. In my case, it's under /home/shared (so it is under /home as you suggested), but it fails with the same error:

pressure-vessel-launch[13106]: Failed to start command: Failed to change to directory '/home/shared/Games/SteamApps/common/Metro Exodus' (No such file or directory)

[smcv]

@LubosD: Please open a separate issue with the full details. This issue is specifically about /usr/local.

[Trezamere]

Does bwrap not like it being in /usr/local or something?

Almost! bwrap doesn't care which path we use, but the pressure-vessel container-launcher (which is what runs bwrap) replaces /usr with the runtime that is used to run Proton. This is how the container runtime can make sure Proton gets the correct libraries. /usr is mounted read-only (to avoid accidentally changing the runtime, which would lead to unpredictable bugs that are difficult to solve), and can't have arbitrary things from the host system mounted below it, so the container setup code treats /usr as reserved, and ignores attempts to share filesystems below /usr with the container. The container setup code is borrowed from Flatpak, which has a similar design and the same restriction.

As a result, /usr/local/games/Steam/steamapps doesn't appear inside the container, therefore /usr/local/games/Steam/steamapps/common/Torchlight doesn't exist, and when we ask to change to that directory, it can't work.

Your real /usr does appear inside the container at /run/host/usr, but, again, it's read-only. Also, if we used that path, games and Proton would be likely to save configuration files that contain /run/host/usr, which wouldn't work when not using the container runtime.

/usr/local isn't a particularly great match for a Steam library, because /usr/local is designed to be system-wide, shared between all users, and "usually" accessed in a read-only way (the only writes into /usr/local are when the sysadmin installs or updates locally-installed software as root), whereas Steam libraries are owned by a particular user and are "usually" read/write to the user that owns them.

I'd recommend putting Steam libraries on a FHS system below /home, /srv or /media, or maybe in /opt or a non-FHS top-level location like the /data you used to work around this. All of those paths should be fine for pressure-vessel.

It would probably be possible to make a special-case exception for /usr/local, but the more special cases we pile into pressure-vessel, the more complex and less robust it gets. So I'd prefer to spend our "complexity budget" on things that are harder to work around, like making it work on more operating systems and under Flatpak.

Making pressure-vessel fail with a clearer warning/error when asked to share a path below /usr would probably be a good thing to have, though.

@smcv I'd like to revisit this as /usr/local is a very common directory for system specific files and at best it's going to be contentious to dictate and assume it needs to be read-only. Steam may be "user specific" but that doesn't mean it can't share install folders between different users, and if you have multiple people using a desktop there is no better place according to the FHS to install games to be shared among everyone than /usr/local/games.

I think if steam is expecting to write into the games install directories as part of its compatibility layer (which are typically read-only on windows as well, and games write to %APPDATA%, My Documents etc.) that is a deficiency in steams behaviour and end-users can't be expected to know this ahead of time. Why does it not run / write the prefix / compat data out of /var or ~/.var, for example?

Finally, since pressure-vessel seems to be based on flatpak, I'd like to use the fact that flatpak does not make this same assumption and only mounts it read-only if explicitly set by the user, https://github.com/flatpak/flatpak/blob/master/common/flatpak-exports.c#L380

And since I'm here, thanks for all the work on making linux gaming possible and for 5.13! I long thought using containers to provide portable runtime environments for games was the solution to the growing and seemingly unmaintainable list of game-specific hacks in upstream application. We have dozens of different production runtime environments at work and the only way to sanely maintain all the dependencies for building is to setup app specific container build environments. It works really well.

[smcv]

I'm going to flag this as a known issue in documentation at some point. I don't know whether it will eventually change, or whether it will always be something that is documented as not expected to work, but either way it is not likely to be something that will change soon, because we have other things that need to be worked on as a higher priority.

at best it's going to be contentious to dictate and assume [/usr/local] needs to be read-only

I think you're mixing up two separate things here.

The main thing stopping us from using a Steam Library below /usr is that the runtime system that we borrow from Flatpak reserves /usr (and /app, and a few other directories) for its own purposes. The host system doesn't get to control the container's /usr, because the whole point of what we're doing with the container is that we want to use the Steam Runtime - not the host system's /usr - as the container's /usr.

We do mount the host system's /usr in the container too, but it's at the container path /run/host/usr rather than the container path /usr - and that's no good if we want to run a Steam game like /usr/local/steamlibrary/steamapps/common/Half-Life/hl2.sh, because Steam will tell us to run paths below /usr/local/steamlibrary, not /run/host/usr/local/steamlibrary. We can't just go through the command-line (and configuration files?) rewriting paths, because the command-line is opaque to the container layer: the meaning of each command-line argument is only known to a lower layer (Proton, if used, and then the game itself). It would be wrong to rewrite an argument just because it happens to look like a path, if it's semantically something else.

Yes, we also do hard-code the host's /usr to be read-only in pressure-vessel - we do the equivalent of Flatpak's --filesystem=host-os:ro and it isn't configurable - but that's not the problem here. The problem is that it isn't mounted in the same place, because the purpose of the runtime system is to make the container's /usr be different.

Flatpak is the same in this respect: if you run a Flatpak app with flatpak run --filesystem=/usr/local/games (or any other path below /usr), that parameter will just get ignored. If you take a look at the commit history of the Flatpak code you linked to, you'll see that I'm in its recent history.

Now, as I said before, obviously we could pile on more special cases to make paths below /usr/local possible - it's all just code, and anything is possible in principle. However, the more complexity we add, the less reliable this is all going to be, and we have higher-priority places than /usr/local to spend our "complexity budget".

A secondary issue is that the container's /usr (the runtime) is mounted read-only, because it is not meant to be modified, which means the container won't have a /usr/local mount point unless we create one in the runtime before entering the container. This is because we're reusing code from Flatpak, which never wants to modify its runtimes, because it stores them in a content-addressed way that would be invalidated by modifying them. We could pre-create mount points before entering the container, or we could ensure that /usr/local is part of all our runtimes and mount a read/write tmpfs onto it; but, again, complexity budget. Flatpak doesn't do that, and we want to minimize how much we diverge from Flatpak in the code we share with it, because that minimizes the number of wheels we have to reinvent.

We also want to minimize the number of paths that always have to end up read/write, because a future goal for the container runtime system is to prevent games from writing to locations other than their intended data paths, so that Steam can know every location that might be needed for cloud sync - analogous to what you said about only writing to %APPDATA%. The only way that's going to be reliable is if new games get developed and QA-tested in an environment that is at least as strict as the environment in which they will eventually be run.

Steam may be "user specific" but that doesn't mean it can't share install folders between different users

Steam developers would know better, but my understanding is that Steam expects to be able to write to all of the Steam libraries that are configured for it.

If you run programs from a directory that is owned or writeable by another user, then that other user can edit the contents of the directory to execute arbitrary code with your privileges (privilege escalation), so this is only something you can do if every user sharing the Steam library trusts every other user who has write access to it

I think if steam is expecting to write into the games install directories as part of its compatibility layer (which are typically read-only on windows as well, and games write to %APPDATA%, My Documents etc.) that is a deficiency in steams behaviour

Individual games might either write to a per-user location, or to their own directory in the library. Both models are common, and Steam does not control what they do. I personally think they should normally only write to %APPDATA% for Windows games or to the XDG basedirs for native Linux games, but it isn't my decision - I don't get to choose how game developers write their games.

Steam compat layers (which are installed alongside games) also write to their installation directory. Proton has a separate directory for mutable state (mainly the WINEPREFIX), but it still writes to the Steam Library to unpack Wine etc. from a tarball (it's distributed that way because the Steam CDN is designed for game assets, so it's better at dealing with a few large files than a lot of small files). The SteamLinuxRuntime machinery works similarly, and it currently doesn't even get a separate directory for mutable state (the one that Steam provides is "owned" by Proton, not by the container runtime) so it basically has no choice but to use its installation directory.

[LubosD]

@smcv You could at least bind map the whole /home area and not just $HOME...

[smcv]

You could at least bind map the whole /home area and not just $HOME...

That's orthogonal to this issue, which is specifically about /usr/local. If not all the required locations below /home are being bind-mounted, please open a separate issue with full details.

[mcronce]

If this is going to remain the case, Steam should throw a very clear UI error when you attempt to install a Windows game on Linux into a library under /usr. As it stands right now, here's how it goes:

• Try to launch game
• Game silently fails to launch, with zero indication that anything went wrong
• User is required to Google to figure out how to get stdout/stderr from games launched by Steam
• User is required to Google just the right line from the output to find an explanation of the problem

Adding documentation only changes the very last step. If it's well-known to Steam's dev team that it is not possible to launch games in a library in /usr, it shouldn't allow users to install there; it should fail with a very clear indication that the library needs to be elsewhere. Personally, I'd probably go as far as to not allow people to put libraries there at all, but if Linux-native games work fine there (do they?) I can see an argument one way or the other.

Either way, "document it on a webpage somewhere" is not a solution.

[smcv]

If this is going to remain the case, Steam should throw a very clear UI error when you attempt to install a Windows game on Linux into a library under /usr

The Steam Runtime doesn't have any user interface other than logging, and doesn't have any control over where you install Steam games/apps/tools, so this would need to happen in the Steam client, not the Steam Runtime. I've opened ValveSoftware/steam-for-linux#10096 for that.