Custom DXVK versions

[zany130]

System Information

• SteamTinkerLaunch version: steamtinkerlaunch-v14.0.20230729-1
• Distribution: Garuda Linux

Feature Description

Found this interesting reshade shader which requires a custom DXVK build

Thought it would be interesting to have the option to set a custom dxvk version following these instructions https://github.com/doitsujin/dxvk#how-to-use

[sonic2kk]

I think this is a great idea, I had a similar idea for D8VK a while back.

The trouble is that afaik, Proton symlinks a lot of DLLs from its lib folders (check the folder structure of ex Proton 8 or GE-Proton). This keeps prefix folder sizes down but makes it tricky to replace certain DLLs as I believe they get overridden. As far as I recall I found the same issue when trying to add D8VK in Winetricks (worked with Wine-GE in Lutris with Touhou 7, but not with Morrowind from Steam with Proton as the DLLs got overridden). I think this is also part of the issue with implementing #794.

So configuring custom DXVK versions might be challenging. Moreso than what is described in the DXVK how-to, though those instructions should work for the likes of Wine-GE.

I'll need to do some more investigation, including checking if what I'm think of the DLL symlinking applies to Proton. I'll also check how Lutris and Heroic allow for this with Proton, if they do at all.

I think it's a great enhancement idea and something I'd love to even extend to vkd3d, and the aforementioned D8VK. Implementation might just be slightly tricky, as I think directly modifying the Proton files is a bit hacky. A potential solution could be to make a copy of the Proton version and mark it as ex <Proton name> - <DXVK version name>.

I'm not home at the moment but I'll take another look at this when I have some time because I've had this idea as well and would really like to see it in STL.

P.S. - Moreso a note to myself but for implementation because this could be extended generally, a generic function to replace certain DLLs could be a good idea. Especially since if we find a way to do this the process should be the same, just the DLLs and maybe paths would differ slightly.

[zany130]

well there is already a .dll override function (which btw couldn't find in the wiki was trying to link to that)

I think that may be expanded upon for this. As far a I can tell overrideing dxvk is as simple as

cp x64/*.dll $WINEPREFIX/drive_c/windows/system32
cp x32/*.dll $WINEPREFIX/drive_c/windows/syswow64

and then setting override I guess its as simple as

d3d11=n,b, d3d10core=n,b, dxgi,=n,b and d3d9=n,b

[sonic2kk]

When launching a game in the prefix, I think Proton will automatically overwrite any DLLs that are not symlinks if they have a symlink for that DLL in its lib folders. I am testing that now though using DXVK HUD to show the DXVK version.

I think setting the d3d11=n,b etc options are for choosing to use the DLLs in the prefix versus those in the folder with the game EXE (ex: some games uses dinput8.dll for mod injection, and you use WINEDLLOVERRIDE="dinput8.dll=n,b" to tell the game to use the dinput8.dll in the game folder instead of the one in the prefix).

[sonic2kk]

I confirmed that when launching a game, Proton will overwrite the DLLs in the prefix if they don't match the ones in the Proton lib folders. It won't overwrite them as a symlink, at least GE-Proton8-9 didn't, but the last-modified time changes to "Just now" (it doesn't if you leave the DLLs as default).

I also checked with DXVK HUD and the version will not change to the version I pasted into the prefix (DXVK v2.1).

---

Proton has a <Proton_Version_Name>/files/share/default_pfx (or <Proton_Version_Name>/dist/share/default_pfx for Valve Proton) which is used to create a template prefix. This has a standard prefix structure like you'd expect from a Proton prefix for a real game. The DLLs in this default_pfx/drive_c/windows/syswow** are a mixed bag, some are standard DLL files and others are still symlinks, including the D3D (DXVK) DLLs. The actual DXVK DLLs stored by Proton are a couple of places:

• syswow64's DLLs are in <Proton>/files/lib/wine/i386-windows/d3d*.dll
  • There is also a folder in files/lib/wine/dxvk which has presumably the 32bit DXVK DLLs as well, but the files in the above folder are not symlinks, so the true files are in the above folder.
• system32's DLLs are in <Proton>/files/lib64/wine/x86_64-windows/d3d*.dll
  • Like syswow64, there is a files/lib64/wine/dxvk folder which presumably has the 64bit DXVK DLLs, but these are not symlinked so the true files are once again in the above folder.

So unfortunately overriding a Proton version's DXVK would probably involve overriding the version in the Proton version files. STL could do this at boot and "clean up" and restore the original DXVK versions on shut-down, but this is not 100% reliable.

The above would also apply to overriding any DLL that Proton has in its default_pfx folder.

[sonic2kk]

I tested with Lutris using HoloCure and Proton 8.0-3c and GE-Proton8-12, and DXVK 2.0, and in all cases I could test, Lutris used the correct DXVK version.

I checked the prefix and Lutris is symlinking the DLLs in the game prefix with the ones in its download directory (~/.local/share/lutris/runtime/dxvk). When changing DXVK versions, the symlinks paths are changed. It does this for both the system32 (64bit) and syswow64 (32bit) DLL folders in the prefix.

After this I did a bit more digging and found that the proton script, which comes with each Proton build (but not builds like Wine-GE) actually has some logic to overwrite updated DLLs with its builtin script. So it's the actual script here doing the updating of the prefix, probably in its update_builtin_libs method under the Proton class.

The proton script is a Python script used to run games with Proton, but the standard wine executable can be used in place of this, though it is not really a good idea to launch Steam games this way (as it applies various Steam game workarounds (see the default_compat_config method). My guess is that Lutris does not use this and so does not have to worry about DLLs being overwritten when using Proton, instead it would just directly run the wine binary.

---

I am not sure if there is much SteamTinkerLaunch can do here to overwrite the DXVK version then. We would need a reliable way to manually update the DXVK version first of all. One potential solution would be to put some DXVK DLLs in with the game exe, but this has a whole other host of problems (some games put DLLs here, not to mention finding the path to the game EXE to begin with and also allowing an override option like we do for ReShade. Speaking of ReShade, this would entirely conflict with it. So it isn't really feasible to do this approach and enforce a DLL override).

I will leave this issue open in case there is another kind of fix, but as it stands right now, this has a similar issue to #794 - We would need to update DLLs in a Proton version's class, and that is probably not a good idea.

As a small side-note: Lutris does allow for launching Steam games, but this just runs the Steam command with the associated AppID (i.e. steam://run/825630) and doesn't pass any of its configuration options to Steam as far as I'm aware.

[zany130]

yeah it seems like there is only two ways of changing the dxvk version for a game running through proton

https://www.gloriouseggroll.tv/compiling-the-latest-version-of-dxvk-on-ubuntu-for-use-with-steam-play/

1. you replace the bundled proton dxvk files (ex: ~/.steam/steam/compatibilitytools.d/GE-Proton7-44/files/lib/wine/dxvk) this will override it for all games that use this proton version

2. Placing it in the game's directory and adding the wine overrides the same way you would for reshade. this has the advantage of working per game but, it can interfere with mods just like reshade would.

An idea I just had what if we create a "custom" version of proton when the user enables the use of a custom dxvk

this custom version could have the specified .dll in place, but otherwise be identical to the original proton version. Though how would we get those dlls?
maybe there could be a dxvk_override folder were a user would place there custom dxvk versions?

just brainstorming here could be completely off base lol

[sonic2kk]

An idea I just had what if we create a "custom" version of proton when the user enables the use of a custom dxvk

This is indeed one solution, and I had a similar idea in my first reply:

A potential solution could be to make a copy of the Proton version and mark it as ex <Proton name> - <DXVK version name>.

The naming doesn't have to be exactly this though 😛 But probably some custom Proton version would be required. We would then store this internally in the STL config files, probably at ~/.config/steamtinkerlaunch/proton/custom. This would copy the selected Proton version and allow us to set the DLLs in the Proton lib folders, which would then get copied to the prefix (and overridden again when switching back).

For how we would handle the DLLs, I'm thinking it would be a symlink from some downloaded STL files in ~/.config/steamtinkerlaunch/downloads, and then we could have DXVK and so on in here. Then, we would rename any existing DLLs we want to override in the copied Proton library folder, and finally create the symlinks. So for example an existing d3d9.dll would get renamed to d3d9.dll.bak or something (consideration here will be important to make sure we pick a name that will almost certainly does not and will never exist).

In terms of making this a generic implementation, we would probably want to name the custom version something like <Proton Name>_STL_DLL_override. My thinking being that we can re-use the same modified Proton version for other DLL overrides, such as vkd3d or LatencyFlex. Each Proton version will have its own copy each time the user selects one, but bugfixes to existing Proton minor versions would be ignored (e.g. proton-8.0-2 would be the folder name but we would ignore proton-8.0-2c/proton-8.0-2d etc).

The way we create the custom Proton versions will probably follow a similar pattern and perhaps even naming convention as the compatdata redirect. We will also need to document that when overriding DLLs a copy of the Proton version is created.

---

The biggest problem with this approach is that Proton is big!! A standard Proton release is around 400mb. On devices like the Steam Deck, consider a user who has a 64gb Steam Deck. This is going to get big pretty quickly!

There may be a possibility of creating a custom Proton version that deeply symlinks from the base version, such as essentially symlinking all of the default_pfx etc and renaming the existing DLL symlinks we want to override. This could be pretty damn complex but it may be possible and could be a good way to keep file size low. I would need to test with manually doing something like this to set a DXVK version.

Every single file would need to be a symlink, which would be... chaos 😅 But it may be possible! And if it saves hundreds of mbs it may be worth it.

This would work because we're creating a symlink copy of the original Proton version, but then overwriting the files only in the copy, which is isolated for use with SteamTinkerLaunch. That way we avoid directly modifying anything related to the base Proton version, which could be used outside of STL. We'll launching using this symlink copy though, with updated DLLs, which would then be the DLLs that get put into the game's prefix. Essentially doing this gives us a safe Proton symlink that uses the original version as a base, but that we can override DLLs as we please within. And since we're using DLLs, the only file size increase we would have to worry about here are the sizes of the DLLs themselves, which should be in the order of 10mb or so; extremely negligible in my opinion, and since we're symlinking the file size is only relevant for the download, as we only need to store each DXVK version once in the downloads folder.

---

This is going to need some manual testing and experimentation from my side, as doing something manually is always a good idea to proof this out. There may be something preventing this from being possible, not to mention this goes on the assumption that the base Proton version will always be available, though if it isn't we should be able to gracefully recover when choosing a new Proton version. And if the selected Proton version no longer existed, we wouldn't be able to launch the game to begin with.

Another case we'll need to handle is ensuring cases where:

1. User installs Proton 8.0 and sets DLL override.
2. Proton 8.0 is then moved for any reason to another location, but still exists.
3. We can probably handle this by re-creating the symlink Proton version and then re-overriding the selected DLLs.

[sonic2kk]

As a quick update I actually tried this out manually last night and problems arose when trying to symlink symlinks. There's probably a way to automatically re-point those symlinks from trying to be a symlink of a symlink (e.g. Proton lib -> default_pfx -> Proton copy for STL) to being a regular symlink that points to the true location (e.g. Proton lib -> Proton copy for STL, cutting out the middle man).

I'll investigate more when I have some free time.

[sonic2kk]

Nevermind, I'm not sure what went wrong last night but I rough set of commands to symlink a Proton version. I have not yet tested if overriding DLLs works as expected but essentially this script will create a deep-symlink copy of a given Proton version (or any directory, really). This could easily be turned into a generic function.

Once we have this copy, anytime an option like a custom DXVK version is used, we'll use this deep copy. We'll rename the existing symlinked file and then put in the overridden DLL. We should probably also "undo" this when the option is disabled. So for DXVK here's an example set of steps:

1. User enables custom DXVK version
2. STL will download and extract this
3. If no existing Proton copy for the chosen version exists, create one
4. If a DLL with the same name as the one we want to override exists, rename it to something like d3d9.dll.org
5. Link the new DXVK DLL into the deep-copy

As well as this, we should also perform some sanity checks somewhere along the way:

1. If we're launching from an overridden Proton version, check if STL has any renamed DLLs (such as d3d9.dll.org)
2. If we do and if there's no equivalent DLL in the same directory (such as d3d9.dll.org but no d3d9.dll) we should handle this somehow
   • This check could end up being expensive and make launch take longer, so we should consider just logging/showing a notifier in cases where the Proton version is missing some DLLs.
   • We should at least log when cases like this happen.
3. If there are no missing DLLs, continue as normal
4. If no options that would override DLLs (such as no custom DXVK version) are enabled, then we should just use the normal Proton version (e.g. Proton 8.0).

An actual happy-path implementation may be more straightforward than I thought, but handling edge-cases where things can go wrong may require some more planning!

[sonic2kk]

Damn, unfortunately many of the symlinks still point to the original Proton directory. So even if we override the DLLs in the lib folders, the ones in default_pfx won't point to them, they'll still point to the ones in the original Proton source directory. There is probably a way to get them to point to the equivalent symlinks in the deep-copy but damn this is going to be very tricky to do...

Removing default_pfx will get the Proton script to re-create it, but then games just crash. Also, these files in default_pfx are no longer symlinks but real files, which is a problem. It's probably generating them using Wine proper instead of the pre-packaged Proton ones, which could be causing the crash.

There may be a way to find and re-create the symlinks in default_pfx, but this is going to get very complicated very fast I feel like...

[zany130]

yeah this is getting kinda complex. I think it might be easier to forget persing an original copy and instead installs/uninstalls a custom dxvk version directly into the selected protons files (similar to wine tricks but for the proton version instead of a single game prefix)

so something where if the user enables a custom dxvk it copies a .dll from a certain directory and installs it into the curre working proton's files and backups up the original .dll for restoring

then disabling it restores the original .dll?

[sonic2kk]

This is a valid solution, and would be feasible to implement, but I have a number of concerns with it.

I am not overly comfortable directly modifying Proton files. If a user enables a custom DXVK version and we override the DLLs, we would temporarily rename the original DLLs. That's fine, but this will affect that Proton version globally until the game with the overridden DLL is closed, and we restore the DLLs. This is a problem for a few reasons. I'll use DXVK 2.0 in these examples with Cookie Clicker and NieR:Automata as sample games, for no other reason than I like those games ;-)

---

If a user has no games running and they use STL to launch a game with DXVK 2.0, for example, then while this game is running, every game launched after will use DXVK 2.0. If I override NieR:Automata to use DXVK 2.0, and then launch Cookie Clicker with the same Proton version, Cookie Clicker will also use DXVK 2.0.

If a user has no games running and they launch Cookie Clicker normally through Steam with Proton, then they launch NieR:Automata with the DXVK 2.0 override for the same Proton version as Cookie Clicker, that will modify the Proton files for the currently running Cookie Clicker, which could be problematic. Since the DXVK DLLs are symlinked to the Proton files, overriding those at runtime for an already running came could potentially cause undesirable behaviour. I'm not sure what the behaviour would be but I don't think changing DLLs at runtime is a good idea.

A similar case: A user has no games running and they use STL to launch NieR:Automata with the DXVK 2.0 override, then they launch Cookie Clicker with the same Proton version. After speedrunning NieR:Automata they close the game and wind down by playing some Cookie Clicker. When they close NieR:Automata which is overridden to use DXVK 2.0 with that Proton version, once again DLLs will be modified at runtime.

So the main problem cases I can foresee are:

• Run overridden game, then run another game with the same Proton version, and it will have all the same Proton DLL overrides
• Run a game with no DLL overrides and then run a game with a DLL override, both with the same Proton version, and you'll get DLL overrides at runtime (since the DLLs are symlinked)
• Run overridden game, then run another game with the same Proton version, then close the overridden game, and you'll also get DLL overrides at runtime

Running two games at once may be an uncommon use-case but I do it enough that it would get in the way for me at least. I can't say how often I've done it with the same Proton version because it's not really something I've thought about, which means it would end up being unexpected behaviour.

A valid response to this (other than "why would anyone play Cookie Clicker instead of NieR:Automata?!") would be to simply not use the same Proton versions, or pick your overridden Proton version carefully. I still don't think that's desirable though, personally. I'd like overriding DLLs to be very low-touch and very easy to revert.

The symlinking idea I had could get around this by using specific Proton symlinks for each game currently running and clearing them afterwards. So each game would have a separate Proton copy that it can use to only have the needed symlinks. But the problem is that this would be incredibly difficult to do.

---

Another broader problem than above with directly modifying Proton files is actually reverting back to the original DLLs. In a happy path that can happen no problem, if nothing goes wrong and games start up and close cleanly with STL, then it'll work as expected. But if a game crashes, or STL crashes, or their power goes out - basically, something goes wrong - users will end up with overridden files in their actual Proton files. This is a very undesirable situation, and it will be bound to happen.

DLLs are overridden in Wine prefixes all the time, by the likes of Winetricks, but there are a few key differences:

• Proton/Wine can attempt to fix a prefix
• It is substantially easier to copy needed files from a prefix and then re-create it, whereas with Proton you would need to re-download the Proton version/verify integrity of game files if using an official Proton build
• Prefixes are much more "transient" than Proton files, it is not uncommon for prefixes to get cleared but it is quite uncommon to need to reinstall a Proton version proper.
  • If this did happen, reinstalling the Proton version is a pain, but then re-running an untouched Proton build would fix up the prefix DLLs and any needed files. Proton can and does often re-create prefixes from scratch on version upgrades and simply moves files from the older prefix into the new one (some compatdata pfx's have corrupted in the name for this reason).

Essentially, it would be a big headache if the overridden DLL files didn't get cleared. We could add a command to recheck a Proton version/all known Proton versions in ProtonCSV.txt but that is more hassle.

The Proton symlinking method could get around this by removing the "ghost" Proton version (that is made up almost entirely of symlinks, save for the overridden DLLs) altogether after a game is closed. And if a game didn't exit cleanly it's ok, because if a user ever runs a game without using the DLL overrides, the overridden "ghost" Proton version is entirely isolated from every other execution. And if the user runs the same game that didn't exist cleanly with a DLL override, we can just remove and re-create the Proton version on launch.

---

So in short, the main problems with directly modifying the "real" Proton files are:

• Simultaneous game launches with the same Proton version will be problematic, as the DLL overrides are not isolated to specific games
• If something goes wrong, the original DLLs are not guaranteed to be restored and a user would probably have to re-download an entire Proton version
• Not a show-stopper but just as a personal opinion, I am not too comfortable with having Proton files directly modified like this, as I see the Proton "installation" itself to be much firmer than a Wine prefix, which I consider to be quite transient. But again that's just my opinion, the first two are the "main" issues.

If creating the "ghost" Proton version with all of the symlinks proves to be too difficult (and it probably will, because I have no idea how one would even set up relative symlinks in the way we would need) then creating a hard-copy of the Proton build itself would be a viable alternative, just at the cost of almost half a gig (or 1-2gb if using Proton-tkg!) per game execution. If Proton is installed to a slower drive, it may also take a significant amount of time to copy it over to the boot drive at ~/.config/steamtinkerlaunch. Hell, it may even take a significant amount of time to transfer 1gb between ~/.steam/root/compatibilitytools.d and ~/.config/steamtinkerlaunch. This also isn't accounting for users who may have exotic setups where their compatibilitytools.d is a symlink and located elsewhere, or who may be daring and have their STL config stored elsewhere. I don't have actual figures but it is faster to create symlinks than copying, which was another advantage to the symlinking method.

But, hard-copying the Proton build would be a viable solution, if implemented correctly. I would rather go with the symlinking solution and that will be the approach I'll try to find a way to go with (even for my own personal use outside of STL) but I don't hold much hope with finding a way to create relative symlinks, without having to go as far as emulating how Proton does it.

• The problem is that the files in the Proton libs point correctly to the source Proton version, but the ones in the default_pfx point to the files in the source Proton's lib and not the one in the symlink Proton's lib.
• There may be a way to get the Proton script to correctly point the symlinks if we remove default_pfx but I did a couple of tests and couldn't quite get it to work properly. Other symlinks may still not be pointing to the correct place, either. We would want as many symlinks as possible to point to the symlinks in the new Proton folder and not to the ones in the source Proton folder.
• Yeah, my head hurts too...

---

I will continue thinking and try to find a solution at some point :-) I appreciate all of your input, and would also like to see this implemented, so this isn't at all me shooting down your ideas or the idea itself.

[zany130]

No those are all fair points. I didn't think of the multiple-game launch. overriding .dll in a game prefix/ install folder is one thing but modifying a proton install affects more things and is more difficult to reverse, So I agree it needs to be done with more care.

If I come up with any more ideas I'll post them here.

[sonic2kk]

Sadly after a bit more investigation, it looks like creating a hard copy of a given Proton version is the only way to go. I'll have a think about any ways to make this more "efficient" - such as on slower drives or copying between e.g. Valve Proton versions installed on the SD card library folder -> STL config folder.

To ease the process of installing into the relevant folder though, we could copy the old DXVK install script. It was removed but it should work if we fetch it with DXVK, or at worst host it on SteamTinkerLaunch-Tweaks, and use this to install the relevant DXVK version.

---

If this is the route that is taken, perhaps a more general "Create Custom Proton" interface could be added to SteamTinkerLaunch, allowing for customising various DLLs and not just limiting this to DXVK. It should also be possible to extend this to Wine versions since they don't have the same kinds of linking that Proton versions have.

If we reframe this feature as the creation of a custom Proton version it would help justify the increased storage penalty. If in future there is a method to more simply creating the required symlinks between Proton versions, we could go with that. I did just have an idea while writing this of only symlinking the non-symlinking files, and seeing how much the Proton script can take care of for us, but I feel like sometime in the past (long before this issue) that I tried this and it didn't quite work.

[sonic2kk]

Possibly related to your original use-case around ReShade @zany130, a project you may want to keep an eye on is vkShade, which aims to implement ReShade shader support on top of Vulkan. That doesn't devalue the original proposal, which is generally useful for custom DXVK DLLs or even custom Proton versions with specific DLLs, but just something I thought you may find interesting in the meanwhile :-)

[sonic2kk]

I unfortunately don't think this is really possible. It would be very hacky to implement because we'd have to replace and restore the DLLs, and for regular Proton this could become problematic.

The Proton symlink copy idea didn't lead anywhere, I tried quite a few approaches since last year and none of them ever really worked.

If someone figures it out I would happily review a PR, but I think for now there is no good solution for this. ☹️