Steam Deck Flatpak can no longer be launched within overriden Launch Options command string from a game due to pinned_libs_64/libcurl.so.4

[bangstk]

This might be a more niche case but here it is.

For some time, 'flatpak' that comes on Steam Deck can no longer be executed within an overridden Steam game properties Launch Options command string (one with %command% in it).

An example of this use case is to launch a flatpak MIDI synth before launching a game under Proton, so that MIDI music would work on Steam Deck for that game, by pasting this in the game launch options: flatpak run org.rncbc.qsynth & %command% ; killall -9 qsynth

Doing so used to work a few months ago but now results in this error, seen if output is redirected to a log:
flatpak: /home/deck/.local/share/Steam/ubuntu12_32/steam-runtime/pinned_libs_64/libcurl.so.4: version 'CURL_OPENSSL_4' not found (required by /usr/lib/libappstream-glib.so.8)

This is fixable by adding LD_PRELOAD=/usr/lib/libcurl.so.4 at the start of the launch option command string but I wanted to report this in case it is something unintended.
So most likely either the Deck version of flatpak or the pinned_libs_64 libcurl.so.4 has been updated to a version the other is incompatible with.

Steam OS Version 3.4
Steam OS Build 20221214.100
Kernel Version 5.13.0-valve35-1-neptune

Steam Version 1671501168

[RyuzakiKK]

When you set a game to be launched via Proton, the whole process will be wrapped inside a pressure-vessel container with the Soldier runtime.
The Flatpak executable that you are calling comes from the Steam Deck OS (Arch Linux based), so it was not meant to be executed in the Soldier runtime. This means that it was just by chance that flatpak run was initially working.

Only games and tools (e.g. Proton) compiled against the runtime are expected to work.

I guess that the easier and cleaner way to achieve what you are trying to do is to use the multitasking capability of the Steam Deck. You can do it by adding Qsynth as a standalone non-Steam game. Then you can execute Qsynth, press the STEAM button, go to Library and launch your game too.

[smcv]

When you set a game to be launched via Proton, the whole process will be wrapped inside a pressure-vessel container with the Soldier runtime.

Actually, everything in the Launch Options runs in the scout LD_LIBRARY_PATH runtime that Steam has traditionally used. That's because scout is the environment that Steam itself runs in, and is Steam's baseline environment for running things that need to be portable to every OS back to about 2014, like Steam and pressure-vessel.

(If you tried to run flatpak inside the soldier environment, you'd find that it wouldn't work anyway, because there is no flatpak executable in the PATH.)

In the case of Proton games, the %command% contains a command that will "escape" from the scout environment, and then run pressure-vessel to put Proton and the game into a soldier container instead; but if you use tricks like & or $() to run extra commands as a side-effect, those are run in the scout environment. I don't think the ability to use & or $() was necessarily intended by the Steam developers, but it works as a side-effect of how the Launch Options were implemented.

scout is mostly compatible with arbitrary commands from the host system, but a few libraries need to be "pinned" (forcing the use of the scout version) because of incompatibilities that happened several years ago, and unfortunately libcurl is one of those.

There are three ways to avoid the libcurl incompatibility, all of them imperfect:

1. Run Steam with STEAM_RUNTIME_USE_LIBCURL_SHIM=1 in the environment, which enables an extra compatibility layer that can load both the SteamOS version of libcurl and the scout version at the same time, to try to make both scout-compatible executables (like older Steam games) and modern-OS-compatible executables (like flatpak) work. This needs to be done at the OS level before Steam is launched (putting it in Launch Options is not enough), so it's probably not straightforward to do on Steam Deck, but it can be done on desktop Linux. This might become the default in a future Steam Runtime version, but it's sufficiently tricky code that we're concerned about possible regressions, so we don't intend to rush into that.

2. Run Flatpak outside the scout environment: ${STEAM_RUNTIME}/scripts/switch-runtime.sh --runtime="" -- flatpak run ... & %command%. Be aware that everything in scripts/ is intended for developer use and is not guaranteed to continue to exist - although I don't intend to remove it any time soon.

3. What @RyuzakiKK suggested: launch non-Steam executables (like Flatpak) separately, for instance as standalone non-Steam games. When you add a non-Steam game to your Steam library, it is run differently, without using the scout environment.

[edited to add the -- pseudo-argument to switch-runtime.sh]

[awarre]

@bangstk thank you for posting this and @smcv thanks for the explanation. I developed a Steam compatibility tool for my Steam Deck which relies on launching a flatpak. Near the end of December it suddenly completely stopped working. Any other application I can still seem to get to launch, but not flatpaks specifically anymore.

[LeJimster]

@kisak-valve Is this being looked to be resolved for SteamOS 3.5??

Some older games are completely broken or don't work well at all with Proton. I have been using dosbox-staging and boxtron from flatpak but this library incompatibility broke it. Ok, we can use workarounds by using the LD_PRELOAD option as above but its far from a perfect solution. Ideally I would like to see Valve work with the creators of Dosbox/boxtron/Luxtorpeda/ScummVM etc so that Linux native versions of these tools are automatically used and no need to do extensive tinkering.

[smcv]

I developed a Steam compatibility tool for my Steam Deck which relies on launching a flatpak

If you are writing your own Steam compatibility tool, you will need to be aware that for historical compatibility reasons, Steam compatibility tools wrapping a native Linux Steam game will generally start up in the LD_LIBRARY_PATH scout environment, not in the host system's execution environment. If parts of the compatibility tool require a different execution environment, it's up to the compatibility tool to arrange for that to happen (in the case of the SteamLinuxRuntime family of tools, this happens in the various shell scripts). In particular, if they are going to execute arbitrary commands from the host system, those commands will want the host execution environment, and not the Steam Runtime. ${STEAM_RUNTIME}/scripts/switch-runtime.sh --runtime="" -- flatpak run ... would be one way to achieve that.

Meanwhile, if the compatibility tool is running the scripts or binaries shipped by a native Linux Steam game, then you will also need to be aware that native Linux Steam games generally expect to be run from the LD_LIBRARY_PATH scout environment - so if you have previously "escaped" from the Steam Runtime environment, you'll need to re-enter it to run the actual game.

Compatibility tools are all about swapping between different execution environments, so the authors of compatibility tools need to keep track of which one of the various possible environments is needed for which commands, and make sure that each command happens in the right environment - there's no avoiding that.

It is not possible to make the LD_LIBRARY_PATH scout environment fully compatible with the host execution environment, because some libraries that it involves have had incompatible changes since the version that's in the Steam Runtime, and replacing them would break older games.

There is more information about the compatibility tool interface in https://gitlab.steamos.cloud/steamrt/steam-runtime-tools/-/blob/main/docs/steam-compat-tool-interface.md. I am not a Valve developer and do not have access to Steam source code, so that document is not authoritative, but it should be reasonably accurate.

[edited to add the -- pseudo-argument to switch-runtime.sh]

[awarre]

@smcv Thanks for the informative reply! That was indeed my issue and the solution.

To add to smcv's response, in my case this ended up being a more complete solution:

${STEAM_RUNTIME}/scripts/switch-runtime.sh --runtime="" -- flatpak run ...

Reference: https://unix.stackexchange.com/a/11382

[smcv]

To add to smcv's response, in my case this ended up being a more complete solution:

${STEAM_RUNTIME}/scripts/switch-runtime.sh --runtime="" -- flatpak run ...

Yes, for a fully general command, you should pass -- to switch-runtime.sh so that it stops interpreting any subsequent arguments as options, and instead passes them through to the command that it is running (in your case that's flatpak). I've edited my previous comments to add the missing -- in the suggested commands, since it's always safe to add, but not always safe to remove.

This is a per-tool thing, rather than a shell thing: each "adverb" tool that wraps another command needs to decide whether/how it will implement --. I've been trying to be consistent about Steam Runtime tools always accepting it.