Add Poster, Banner, Logo, etc to steamtinkerlaunch add Non-Steam game as launch options

[trentondyck]

System Information

• SteamTinkerLaunch version: N/A
• Distribution: Steam Deck

Feature Description

@sonic2kk Since we have app id, this also unlocks the ability to add poster, banner, logo per app. Would be a nice addition to steamtinkerlaunch:

Working example in bash/python:

https://github.com/trentondyck/horizon_scripts/blob/70bd701f06596ff4f4f2fa96054abc6a7c03a0a5/install-or-update-horizon.sh#L107

Context:

#729

[sonic2kk]

This would be great, but it looks like it relies on Python to get the AppID for the Non-Steam Games? Otherwise nice find!

Also wondering that even though it relies on Python, can it generate the AppID? The discussion in ProtonUp-Qt that I was involved in ended up just parsing it instead of calculating it.

SteamTinkerLaunch is a Bash script and will not use Python in any capacity, so we need a way to calculate this from Bash. It also probably has to be written out to the shortcuts file.

As an aside, if we can generate this with pure Bash, this can also be included in the Add Non-Steam Game UI, which I originally tried and failed to do in a PR several months ago.

(This specifically refers to Non-Steam Games it seems so just editing the title of your proposal to reflect this 😄)

[trentondyck]

It does. Line 104 parses shortcuts.vdf and returns appid.

That's too bad about python, seems like the correct way to parse these files, though it does have the requirement of both python and the vdf module which I assume is what you're trying to avoid.

If we can pick apart https://github.com/ValvePython/vdf/blob/d76292623e326fb165fe3bdb684832cdf30959d4/vdf/__init__.py
maybe there's a way.

[sonic2kk]

It does. Line 104 parses shortcuts.vdf and returns appid.

That sounds like reading, not calculating :-) We need to calculate the AppID, since we are adding artwork etc before Steam calcuates the AppID. Please let me know if I'm misunderstanding here

seems like the correct way to parse these files

Nope, it's also possible with Go and TypeScript :-) There is no "correct" way, Python is just one way. In fact, various parts of STL can parse VDF files in Bash. I just don't really understand how Frostworx did it or how to apply it to this instance. Probably it can't be applied here in this specific case at all since we have to calculate the Non-Steam AppID.

though it does have the requirement of both python and the vdf module which I assume is what you're trying to avoid

Absolutely, SteamTinkerLaunch is a Bash utility. It has various command-line related dependencies but it is focused on being as light on dependencies as possible. I love Python, but it has no place in SteamTinkerLaunch. I don't intend to mix languages and introduce huge dependencies in the process.

Most of what STL depends on will come with a standard rolling distro except for the front-end Yad dependency. Having to introduce Python and on top of that, tell users to install the VDF library (and trying to do this on Steam Deck, not to mention bloating the STL Flatpak even more) will just be a major pain.

If we can pick apart https://github.com/ValvePython/vdf maybe there's a way.

I totally agree! I don't understand how the parsing works even for the VDF library, I did take a look before to try and figure it out but it's not something I'm knowledgeable on. Hopefully someone else can step up and implement it, I really want to see more community development in SteamTinkerLaunch.

[sonic2kk]

I think it was discussed before in the issue you linked, but just to mention again: ProtonUp-Qt author DavidoTek actually figured out how it was calculated by looking at a Go implementation: DavidoTek/ProtonUp-Qt#175 (comment). The full Go code for the Non-Steam Game generation is here: https://github.com/boppreh/steamgrid/blob/9c8788db4f04613ecfb3e8fb36a0af02395e4593/games.go#L105-L137

It seems they parse the shortcuts.vdf file using some regex to grab based on a defined pattern, and then generate the ID using part of that regex they parsed like this:

gameID := fmt.Sprint(binary.LittleEndian.Uint32(gameGroups[1]))  // DavidoTek confirmed that this gameID is the correct AppID

So the solution to the AppID pain in my mind is this:

• Figure out how exactly the regex they create works (https://github.com/boppreh/steamgrid/blob/9c8788db4f04613ecfb3e8fb36a0af02395e4593/games.go#L117)
• Figure out how to convert the regex that the Go library does (it's this function from its regex package, relevant line is here) into Bash, in particular the -1 argument
• Figure out how to do that Go calculation in Bash

Essentially we need to implement that Go function in Bash. I unfortunately have next to zero experience writing Go code and certainly no practical experience with using any sort of packages with it and whatnot, I also have no real understanding of the calculation that's going on here as my maths skills are not the greatest.

If anyone wants to tackle this (doesn't have to be OP), this issue can be left open and used for discussion and collaboration on development. I'll help out where I can and would also be happy to review and PRs that came off the back of this. This is a feature I would like to see added to SteamTinkerLaunch (see #550, #576) but that after a lot of trying I could not figure out with my limited understanding. Back then I was even on the wrong track, it wasn't until DavidoTek/ProtonUp-Qt#175 that I saw they were calculated differently (e.g. without a CRC). I am a bit burned out on figuring this out personally at least for now so I did not look into it all that deeply.

Maybe this background will be useful to someone who wants to add this :-) It is pretty much essential that it be done with Bash, external command-line utility dependencies could be used no problem though adding a dependency like Python or Go is not the solution for STL. Perhaps someone that understands how to do the calculation and regex matching can implement it with grep and maybe some command-line math utility to do the unsigned integer conversion.

[zDEFz]

I found that when applications have already been added, the new settings may be ignored. Thus, I thought I messed up my installment. I had to reinstall steam to cleanup the vdf file. Cause my passed proton ver was ignored. Was the easiest way. By the way, attached is a bash script to process many games at once. All you need is a filelist!
steam.txt

By the way, checking the proton thing does not seem consistent, and I dont know why.

Do you think ChatGPT would be a candidate to "convert" one language into another? I don't have a account on ChatGPT, cause of privacy concerns, but it may be worth a try...

[sonic2kk]

By the way, checking the proton thing does not seem consistent, and I dont know why.

I saw a little bit about that in the discussion on #729, though that isn't really a goal for this feature proposal, it's still useful to know. It is something I had hoped would be possible. It's something that can be explored once the AppID can be generated. To this point...

Do you think ChatGPT would be a candidate to "convert" one language into another?

Heh, ChatGPT is something I find quite fascinating, and I had actually asked it previously to convert between languages (not for any "real" use-case, just to see if it could convert basic programs). I think I only tested once or twice but I think it was able to do it.

If we could get a better understanding of how the calculation is done for the Non-Steam AppID it may be possible to ask ChatGPT :-) Though are projects """allowed""" to use code generated by ChatGPT? I would guess so but I am not 100% sure and would err on the side of caution here.

[zDEFz]

If we could get a better understanding of how the calculation is done for the Non-Steam AppID it may be possible to ask ChatGPT :-) Though are projects """allowed""" to use code generated by ChatGPT? I would guess so but I am not 100% sure and would err on the side of caution here.

Privately, you can ask it to learn how code X works... But corporate / private information does not belong to ChatGPT. It's not forbidden to ask privately. You could gain knowledge, and abstract / change it from there. However, I would never trust ChatGPT, thats exactly why it is a necessity to analyze the output thoroughly. But it may ease the research. It is nice when it comes to explaining code, but not always. I'm also not a fan of ChatGPT answers or Projects. Its just like, you know ... it may provide insights.

[trentondyck]

I used it to parse the shortcuts vdf, super useful! But converting large libraries is not its strong suite. Short bits of code at a time can be accomplished.

[sonic2kk]

I did a little investigation into using ChatGPT to convert the Go parser into Bash, but did not get anywhere, it simply did not read any information properly from the shortcuts.vdf file. The rest of the script looked correct, so I wonder if the .vdf file has to be "opened" in Bash in a specific way (for Python I think it needs the rb flag, perhaps with Bash we need to use something like xxd).

[sonic2kk]

Perhaps if we take the format that we write out to the file we can calculate the App ID for that shortcut based on the data we already have. Maybe if we try to get the AppID from what we would parse from the shortcuts.vdf. If we pass something like this to the parser: "\x01appname\x00${NOSTAPPNAME}\x01Exe\x00${NOSTEXEPATH}\x00", perhaps it could get the Non-Steam AppID.

Not sure though.

[trentondyck]

I don't think you need it, but I found a nodeJS way to calculate steam_id, converted it to python using our best friend and published that in my repo (horizon_scripts). Supposedly there's a way to do it in bash but requires the crc32 utility which doesn't come pre-installed. I think it's unrelated to AppID (Sorry, bit off topic here), but interesting nonetheless:

#!/bin/bash

target='"C:\Program Files (x86)\Origin\Origin.exe"'
label='Fubar Game'

checksum=$(echo -n "$target$label" | crc32)
steam_id=$((checksum | 0x80000000))
top_32=$steam_id
bottom_32=0x02000000
legacy_steam_id=$((top_32 << 32 | bottom_32))

echo $steam_id
echo $legacy_steam_id

wasn't able to test bash version, but python version works nicely.

[sonic2kk]

The CRC32 algorithm is only for Big Picture, and possibly only old Big Picture. Regular Steam apps no longer use the CRC algorithm. See here (https://github.com/boppreh/steamgrid/blob/master/games.go#L115-L137) and this comment by DavidoTek that verifies that the CRC calculation is not correct anymore (DavidoTek/ProtonUp-Qt#175 (comment)).

EDIT: I'm looking into a potential solution for this issue that won't involve calculating the AppID. No ETA, just something I'm investigating.

[sonic2kk]

Still haven't found a way of extracting the AppID using Bash, but with a little help from ChatGPT I did come up with a Python solution that doesn't rely on the vdf library:

import re

pattern = re.compile(b"(?i)\x00\x02appid\x00(.{1,4})\x01appname\x00([^\x08]+?)\x00\x01exe\x00([^\x08]+?)\x00\x01.+?\x00tags\x00(?:\x01([^\x08]+?)|)\x08\x08")  # Taken from SteamGrid Go code
with open('shortcuts.vdf', 'rb') as f:
    data = f.read()

matches = pattern.findall(data)
for match in matches:
    appid = int.from_bytes(match[0], byteorder='little')
    print(appid)

This properly extracts each AppID from the shortcuts.vdf file. It can also get the name, so you can do something like print(f'{match[1].decode("ascii")} ({appid})') which will output Game Name (AppID).

If this can be converted to Bash it should be possible to extract the AppID and then use the newly-implemented Game Art Selection feature to set the artwork. My own attempts including with ChatGPT have come up short, but a PR would be more than welcome from someone that could add a function to do what this Python code does.

[sonic2kk]

I managed to come up with something pretty close by looking at the offsets with a xxd hexdump and getting ChatGPT to lend some assistance!

#!/bin/bash

## CLOSE!!
# Get the AppID field (little-endian unsigned 32-bit integer starting at offset 0x1c)
APP_ID=$(xxd -p -s 0x1c -l 4 shortcuts.vdf | tac -rs .. | tr -d '\n' | xxd -r -p | od -N 4 -tu4 -v | awk '{print $2}')  # CLOSEST
echo "AppID: $APP_ID"

# Get the AppName field (null-terminated string starting at offset 0x26)
APP_NAME=$(xxd -p -s 0x22 shortcuts.vdf | tr -d '\n' | sed 's/00/\n/' | head -n 1 | xxd -r -p)
echo "AppName: $APP_NAME"

This returns an AppID but the correct app name for one entry. I have two shortcuts in this file, but this code just gets the first one.

The AppID it returns is incorrect. It returns 1884184941 but the correct value is 3934526018.

[trentondyck]

Hmm, not sure what the issue is yet but that's a nice start. As you mentioned it seems to print the wrong appID, but the appname is correct for me. it only picks the first entry as well in a list of many--what's making it do that in the "APP_ID=" bash command?

here first two are from your bash script, others a loop from vdf library in python (with correct app ids)

AppID: 1886282093
AppName: EmulationStation-DE-x64_SteamDeck
appname: EmulationStation-DE-x64_SteamDeck, appid: 3647351456
appname: Cemu, appid: 2814663076
appname: Citra, appid: 2986775124
appname: Dolphin (emulator), appid: 2882124603

[sonic2kk]

I think it only returns one, maybe because of the head command? Maybe this could be done in a while loop and for each entry it could run the xxd -r -p command.

To be clear and fair I wasn't trying to get it to do it in a loop yet, and I'm honestly amazed it actually gets the name right for any entry.

For the AppID being wrong it could be that the offset is wrong, I haven't delved too deeply yet. But we're getting somewhere!

[sonic2kk]

The following Bash one-liner will extract all AppNames from the shortcuts.vdf file:

grep -a -o -P '(?i)\x01appname\x00(\K[^\x00-\x08]+?)(?=\x00\x01)' shortcuts.vdf 

Now I need to figure out how to do something similar for the AppIDs. Then it could go in one grep command and we can get the AppID and AppName for a shortcut from the shortcuts.vdf file.

The difference in this approach and earlier is that earlier it was using xxd to get a value from the offset. The first value will probably always have an offset of about 0x22, but later values will have differing offsets depending on other values in the file e.g. the length of the appname and path and so on. So we have to use regex pretty much.

In Python, this regex works, but it doesn't in Bash: (?i)\x00\x02appid\x00(.{1,4}). Likely there's some oddities with however the grep regex engine works that doesn't work the same way as Python's.

Once we get the value we have to convert it from a little endian to a 32bit unsigned integer.

[trentondyck]

ChatGPT is depressing for this complex stuff. this is as far as I got (with a shortcuts.vdf with 3 appids so its printing appid 3 times basically):

xxd -p shortcuts.vdf | tr -d '\n' | sed 's/\(..\)/\1 /g' | grep -a -o '61 70 70 69 64'
61 70 70 69 64
61 70 70 69 64
61 70 70 69 64

as you mentioned anytime we're adding some regex filter to grep ( i.e. {1,4} ) it poops the bed.
Anyone interested in tinkering here's an example shortcuts.vdf

https://drive.google.com/file/d/1cZvQdSCSfVbHu5WmM7drjG4GIqrI3v9V/view?usp=sharing

And the proper 3 appids (expected result):

3266830982
4002098395
3909329041

[zDEFz]

[0-9]{10} gets you 3266830982 out of the given example. Not sure where 4002098395 and 3909329041 are from.

Oh. I get it. I should take the sample as thing to play around in the bash. E.g copy paste the string into a new file and then come up with those 3 results reproducibly. But then again.

What if we ask ChatGPT the following:

Guess the formula used to convert the decimal 3266830982 to the two values 4002098395 and 3909329041. Make a bash script to approximate two 10 digit decimals from a 10 digit decimal in order. Make sure the result is unique but reproducible. E.g you always end up on 4002098395 and 3909329041 if input was 3266830982.

I really hope it doesnt end up in a simple subtraction :D

[zDEFz]

I'm not sure what you mean, can you paste the full command you ran to get 3266830982 as a result? 4002098395 and 3909329041 are two distinct appids and have nothing to do with the first

Just cat shortcuts.vdf | grep -Eo "[0-9]{10}" | uniq which returns 3266830982
The shortcuts.vdf stems from your sample:

^@shortcuts^@^@0^@^Bappid^@<86>â·Â^AAppName^@HorizonXI-Launcher.exe^@^AExe^@"/home/deck/horizon-xi/lib/net45/HorizonXI-Launcher.exe"^@^AStartDir^@"/home/deck/horizon-xi/lib/net45/"^@^Aicon^@^@^AShortcutPath^@^@^ALaunchOptions^@^@^BIsHidden^@^@^@^@^@^BAllowDesktopConfig^@^A^@^@^@^BAllowOverlay^@^A^@^@^@^BOpenVR^@^@^@^@^@^BDevkit^@^@^@^@^@^ADevkitGameID^@^@^BDevkitOverrideAppID^@^@^@^@^@^BLastPlayTime^@-Ûæc^AFlatpakAppID^@^@^@tags^@^H^H^@1^@^Bappid^@Û,<8b>î^AAppName^@windower.exe^@^AExe^@"/home/deck/.local/share/Steam/steamapps/compatdata/3266830982/pfx/drive_c/Windower/windower.exe"^@^AStartDir^@"/home/deck/.local/share/Steam/steamapps/compatdata/3266830982/pfx/drive_c/Windower/"^@^Aicon^@^@^AShortcutPath^@^@^ALaunchOptions^@^@^BIsHidden^@^@^@^@^@^BAllowDesktopConfig^@^A^@^@^@^BAllowOverlay^@^A^@^@^@^BOpenVR^@^@^@^@^@^BDevkit^@^@^@^@^@^ADevkitGameID^@^@^BDevkitOverrideAppID^@^@^@^@^@^BLastPlayTime^@qÙæc^AFlatpakAppID^@^@^@tags^@^H^H^@2^@^Bappid^@<91> ^Cé^Aappname^@Horizon XI^@^AExe^@"/home/deck/horizon-xi/lib/net45/HorizonXI-Launcher.exe"^@^AStartDir^@"/home/deck/horizon-xi/lib/net45/"^@^Aicon^@/home/deck/horizon-xi/icon.png^@^AShortcutPath^@^@^ALaunchOptions^@^@^BIsHidden^@^@^@^@^@^BAllowDesktopConfig^@^A^@^@^@^BAllowOverlay^@^A^@^@^@^Bopenvr^@^@^@^@^@^BDevkit^@^@^@^@^@^ADevkitGameID^@^@^BLastPlayTime^@^@^@^@^@^@tags^@^H^H^H^H

[trentondyck]

(deck@steamdeck horizon_scripts)$ cat shortcuts.vdf | grep -Eo "[0-9]{10}" | uniq
grep: (standard input): binary file matches

[zDEFz]

(deck@steamdeck horizon_scripts)$ cat shortcuts.vdf | grep -Eo "[0-9]{10}" | uniq
grep: (standard input): binary file matches

[zDEFz]

http://ix.io/4qBj

[zDEFz]

ok something is lost in translation when I copy/paste. try it on this file https://drive.google.com/file/d/1cZvQdSCSfVbHu5WmM7drjG4GIqrI3v9V/view?usp=sharing

Try cat shortcuts.vdf | grep --text -Eo "[0-9]{10}" | uniq

[sonic2kk]

Nice to see the discussion going on here :-) I don't have anything more to add (for now anyway) but please feel free to continue discussing here and sharing whatever you find out!

[trentondyck]

Maybe GPT 4 will answer the question for us... lol

[zDEFz]

Maybe GPT 4 will answer the question for us... lol

Whats the current question?

[trentondyck]

Maybe GPT 4 will answer the question for us... lol

Whats the current question?

https://www.youtube.com/live/outcGtbnMuQ?feature=share

GPT 4 Looks amazing, so far I haven't been able to use GPT 3.5 to successfully convert that python script above to bash. Ultimately its the question this entire thread is based on I was just making a remark that if we wait until GPT 4 is live maybe it will translate for us better.

[sonic2kk]

This seems like the kind of problem that in 3-4 years we'll look back on and think "wow, AI was so bad back then" haha.

I have not made much progress with regards to this issue, but just once again this is something I would love to see in STL. Hell, even as a proof-of-concept I think it would be really cool to have this done completely in Bash. I really appreciate all the time and discussion that has gone into this, so I think I will actually pin this issue in the hopes of bringing even more attention to it.

Maybe someone a lot more experienced in Bash and parsing binary files will be able to help solve this :-)

[trentondyck]

GPT4 does seem to do better than 3.5, but still returns the wrong result. example:

#!/bin/bash

file="shortcuts.vdf"

hex_data=$(xxd -p -c 20000 "${file}")

echo "Hex data:"
echo "${hex_data}"
echo

hex_matches=$(echo "${hex_data}" | grep -oP '(?<=617070696400)[0-9a-fA-F]{8}')

echo "Hex matches:"
echo "${hex_matches}"
echo

for hex_appid in $hex_matches; do
  appid=$((16#${hex_appid}))
  echo "Hex: ${hex_appid} Appid: ${appid}"
  echo "${appid}"
  echo
done

Result

./script16.sh 
Hex data:
0073686f727463757473000030000261707069640086e2b7c2014170704e616d6500486f72697a6f6e58492d4c61756e636865722e657865000145786500222f686f6d652f6465636b2f686f72697a6f6e2d78692f6c69622f6e657434352f486f72697a6f6e58492d4c61756e636865722e657865220001537461727444697200222f686f6d652f6465636b2f686f72697a6f6e2d78692f6c69622f6e657434352f22000169636f6e00000153686f7274637574506174680000014c61756e63684f7074696f6e73000002497348696464656e000000000002416c6c6f774465736b746f70436f6e666967000100000002416c6c6f774f7665726c61790001000000024f70656e56520000000000024465766b69740000000000014465766b697447616d6549440000024465766b69744f7665727269646541707049440000000000024c617374506c617954696d65002ddbe66301466c617470616b41707049440000007461677300080800310002617070696400db2c8bee014170704e616d650077696e646f7765722e657865000145786500222f686f6d652f6465636b2f2e6c6f63616c2f73686172652f537465616d2f737465616d617070732f636f6d706174646174612f333236363833303938322f7066782f64726976655f632f57696e646f7765722f77696e646f7765722e657865220001537461727444697200222f686f6d652f6465636b2f2e6c6f63616c2f73686172652f537465616d2f737465616d617070732f636f6d706174646174612f333236363833303938322f7066782f64726976655f632f57696e646f7765722f22000169636f6e00000153686f7274637574506174680000014c61756e63684f7074696f6e73000002497348696464656e000000000002416c6c6f774465736b746f70436f6e666967000100000002416c6c6f774f7665726c61790001000000024f70656e56520000000000024465766b69740000000000014465766b697447616d6549440000024465766b69744f7665727269646541707049440000000000024c617374506c617954696d650071d9e66301466c617470616b4170704944000000746167730008080032000261707069640091a003e9016170706e616d6500486f72697a6f6e205849000145786500222f686f6d652f6465636b2f686f72697a6f6e2d78692f6c69622f6e657434352f486f72697a6f6e58492d4c61756e636865722e657865220001537461727444697200222f686f6d652f6465636b2f686f72697a6f6e2d78692f6c69622f6e657434352f22000169636f6e002f686f6d652f6465636b2f686f72697a6f6e2d78692f69636f6e2e706e67000153686f7274637574506174680000014c61756e63684f7074696f6e73000002497348696464656e000000000002416c6c6f774465736b746f70436f6e666967000100000002416c6c6f774f7665726c61790001000000026f70656e76720000000000024465766b69740000000000014465766b697447616d6549440000024c617374506c617954696d650000000000007461677300080808080a

Hex matches:
86e2b7c2
db2c8bee
91a003e9

Hex: 86e2b7c2 Appid: 2263005122
2263005122

Hex: db2c8bee Appid: 3677129710
3677129710

Hex: 91a003e9 Appid: 2443183081
2443183081

It tried a number of permutations on this but the result was always an incorrect hex or appid or unsigned int. I'll try again later as it's rate limiting us :)

[sonic2kk]

Perhaps the data its pulling needs to be converted in some way. The Python code I have seen had to do something along these lines: int.from_bytes(appid_from_regex, byteorder='little')

[trentondyck]

It's getting so close. GPT4 is able to go backwards and calculate the correct hex from an unsigned 32-bit integer via this script:

#!/bin/bash

# Store your unsigned 32-bit integer in a variable
unsigned_int=3266830982

# Debug: Print the input unsigned 32-bit integer
echo "Input unsigned 32-bit integer: $unsigned_int"

# Initialize little-endian hexadecimal representation
little_endian_hex=""

# Convert the 32-bit unsigned integer to a little-endian hexadecimal representation
# Iterate through the 4 bytes (32 bits) of the unsigned integer
for (( i=0; i<4; i++ )); do
  # Calculate the current byte by isolating the relevant 8 bits
  # using bitwise AND with 255 (0xFF) and then bitwise right-shifting 8 bits * i
  current_byte=$(( ($unsigned_int & (255 << (8 * i))) >> (8 * i) ))

  # Debug: Print the current byte (decimal)
  echo "Current byte (decimal): $current_byte"

  # Convert the current byte (decimal) to a hexadecimal representation (with leading zero if needed)
  current_byte_hex=$(printf "%02x" $current_byte)

  # Debug: Print the current byte (hexadecimal)
  echo "Current byte (hexadecimal): $current_byte_hex"

  # Append the current byte (hexadecimal) to the little-endian hexadecimal representation
  little_endian_hex="${little_endian_hex}${current_byte_hex}"
done

# Debug: Print the little-endian hexadecimal representation
echo "Little-endian hexadecimal representation: $little_endian_hex"

output:

Input unsigned 32-bit integer: 3266830982
Current byte (decimal): 134
Current byte (hexadecimal): 86
Current byte (decimal): 226
Current byte (hexadecimal): e2
Current byte (decimal): 183
Current byte (hexadecimal): b7
Current byte (decimal): 194
Current byte (hexadecimal): c2
Little-endian hexadecimal representation: 86e2b7c2

It just fails to convert the hex back to an unsigned int from this script which it seems to like

#!/bin/bash

# Store the original hex in a variable
original_hex="86e2b7c2"

# Debug: Print the original hex
echo "Original Hex: ${original_hex}"

# Reverse the byte order for little-endian hexadecimal
little_endian_hex=$(echo "${original_hex}" | sed 's/\(..\)/\1 /g' | rev | tr -d ' ')

# Debug: Print the little-endian hex
echo "Little-endian Hex: ${little_endian_hex}"

# Convert the little-endian hexadecimal to an unsigned 32-bit integer
appid=$((16#${little_endian_hex}))

# Debug: Print the AppID
echo "AppID: ${appid}"

output

Original Hex: 86e2b7c2 # Correct Hex
Little-endian Hex: 2c7b2e68 # I don't know what this is
AppID: 746270312 # Should be 3266830982

[sonic2kk]

For what it's worth, I tried running this Bash snippet you posted earlier for for reading the shortcuts.vdf file and it doesn't work on my shortcuts.vdf file:

Snippet:

#!/bin/bash

file="shortcuts.vdf"

hex_data=$(xxd -p -c 20000 "${file}")

echo "Hex data:"
echo "${hex_data}"
echo

hex_matches=$(echo "${hex_data}" | grep -oP '(?<=617070696400)[0-9a-fA-F]{8}')

echo "Hex matches:"
echo "${hex_matches}"
echo

for hex_appid in $hex_matches; do
  appid=$((16#${hex_appid}))
  echo "Hex: ${hex_appid} Appid: ${appid}"
  echo "${appid}"
  echo
done

With two shortcuts in my file, it outputs this:

Hex data:
0073686f727463757473000808

Hex matches:

---

Apart from that, this is getting really close!

[trentondyck]

probably worth posting a link to a google drive for your shortcuts.vdf (copy paste doesn't seem to translate properly)

[sonic2kk]

Oops, totally forgot about attaching the shortcuts file. I'll create a better reproduction file with more shortcuts and attach it once I have some more time for this.

---

I did a bit of testing on this tonight and noticed that Steam overwrites the AppID for a shortcut the first time it's launched. Or it seems to anyway. But when using the AppID written out to the file to set the game art, it still doesn't see it.

Probably instead of parsing I would guess this will still come down to generating the AppID somehow...

[sonic2kk]

I had a thought earlier this week about Non-Steam Game AppIDs. Going back over everything here and retracing a lot of my steps, I had this crazy idea: What if we just wrote out our own AppID to the shortcuts.vdf file? And it worked.

Okay, it wasn't as simple as that, but locally I have a working implementation of generating and writing out a custom Steam shortcut AppID that I'm going to push after I finish writing this. It doesn't yet have any functionality to copy over the artwork, but actually setting and knowing the AppID works, so all that's needed is some UI and logic to take in the artwork. We should also be able to re-use some stuff from #757 to accomplish this, meaning we shouldn't need to re-implement logic to copy over artwork!

I'll try and explain the parameters for generating the AppID, because the answer was under our noses the whole time!

Two useful resources I referred to during this test was the Python VDF library, which I used to read the expected values from the VDF file. The other resource was Integer Encoder, which I used to encode/decode from signed integers, unsigned integers, and hex. This allowed me to check at each stage that my code was matching the VDF file expected values.

---

Steam stores shortcuts in a binary file called shortcuts.vdf, located at /path/to/steamroot/userdata/<userid>/config/shortcuts.vdf. This uses Valve's proprietary binary format, though the format is known and documented in various places - [SteamDB](https://github.com/boppreh/steamgrid/blob/master/games.go#L117, STL's own addNonSteamGame function also writes out with the actual binary format, and finally what was most useful to me, the Python vdf library's method binary_dumps. This prints a string representing the output that would be written to the VDF file, so you can see its actual binary format.

The most useful thing I got from this was, of course, how this file stores the AppID. It's something like this: \xc1\xc2\x5a\xdc.

All of those \x parts are just part of the binary format, so stripping that back, we get c1c25adc. This is a hexidecimal value, and of course for the purposes of this issue (the ability to add Non-Steam Game Artwork when adding a Non-Steam Game), we need an integer AppID.

---

So now we know Steam stores the Non-Steam AppID as hex, but the AppID we need is an integer!

This hexidecimal doesn't represent a regular integer that you might be used to though, it is a signed 32bit little-endian integer. This can be seen by using the Python vdf library to load the shortcuts.vdf file, you'll see your shortcuts have an appid of something like -598031679 (DavidoTek, creator of ProtonUp-Qt, also figured this out in a comment on that project a few months ago). So we need to convert our hex value into a 32bit signed integer. I am not sure how to do this conversion in Bash, to be honest, but that isn't too important, we're just working backwards at a high level. The important thing to highlight is that this is the process to go from the value in the shortcuts.vdf to the AppID we require to set the game artwork.

Once converted, we'll be able to turn our hex that we got from the binary shortcuts.vdf (i.e. c1c25adc) into a 32bit signed integer (i.e. -598031679). But we have another roadblock: Steam AppIDs are positive integers, not negative integers! Sadly this isn't simply a case of removing the sign either, we can't just trim off the - from -598031679. So we need to convert our signed integer to an unsigned integer (such as 3696935617), and bingo, we have our AppID which is used to set game artwork!

---

So to recap, to go from the value in the shortcuts.vdf file:

1. Extract binary hex, including \x's
2. Clean this up to get a clean hex string (ex: c1c25adc) -- This is a 4byte little-endian integer
3. Convert this hex to a 32bit signed integer (ex: -598031679)
4. Convert this to a 32bit unsigned integer (ex: 3696935617)
5. Profit!

Now we know how Steam stores those pesky AppIDs. But how do we write these out, and why did it not work before?

---

The crux of why all of this wasn't working before is because the shortcuts.vdf was getting a positive hex value that was then written out as a Big-Endian integer instead of a Little-Endian. A Little-Endian is basically the hex string but with a swapped byte order, I'm sure there's a lot more to it that computer scientists would scream at me over, but that's what is relevant for this. Essentially, we were just generating a random integer (ex 123456789), converting it to hex, doing the binary formatting to include the \x's, and then writing that out. Steam wasn't happy about this, and would overwrite our AppID.

What we really have to do to get this right, is generate a random 32bit signed integer, and then convert that to a 4byte little-endian hex string which we write out to the AppID. This allows us to set our custom AppID for the Non-Steam Game. But to actually solve the grids issue, we need to then get the unsigned 32bit integer from this base signed integer we generated the little-endian hex from. Once we have this unsigned integer, we can put some artwork into the Steam artwork grids folder using that unsigned integer, and just like magic, our artwork will show up in Steam!

---

I was able to accomplish this, and I have the Bash code to actually generate a Non-Steam AppID that we can know ahead of time, allowing us to eventually set the artwork. This AppID and its other variations (hex, formatted hex, and the raw signed integer) are also logged.

This proposal should be possible to implement now, and I'll look into implementing it as soon as I can :-)

[trentondyck]

nice!

[sonic2kk]

#902 was merged, so the next step is to add the logic for Non-Steam Games to (optionally) take in the artwork. This is probably going to reuse a lot of the set game artwork logic, as well as a bunch of strings and UI stuff from #576. It looks like I started my journey into Non-Steam AppID's just over a year ago because that PR is from September 5th 2022, wow!

I'll also implement the command-line ansg logic to take these flags in, as that work will be included as part of adding it to the UI.

Fingers crossed that this should be straightforward to implement! The hard part (getting the AppID) should be done now.

[sonic2kk]

This feature has been implemented with #903. SteamTinkerLaunch v14.0.20230916-1 or above (8f24849) will now include this feature. I tested it out with a Non-Steam Game (screenshot of UI and moved files in that PR) and it worked. It's pretty useful, I've wanted this feature for a while.

This also applies to the commandline usage as well (addnonsteamgame and the shorthand ansg), because of how the UI portion works it essentially just takes the information from the UI and passes it to the commandline function. All of this then feeds into setGameArt from #757, which takes an AppID as an argument. The AppID we are now able to generate when adding a Non-Steam Game is just passed to this function, as well as the paths to the files selected in the Add Non-Steam Game GUI. So this change was pretty trivial to make!

This opens the door to future possibilities, such as using a specific compatibility tool with a Non-Steam Game and writing it out to the config.vdf (just a text file). The reason we couldn't do this before was that we didn't have the AppIDs. There are still some non-trivial things to solve with this, such as knowing what name exactly to set for each selected compatibility tool, and some logic around checking if compattool set then use compattool otherwise skip. That part shouldn't be too hard though.

Another set of features I'm considering are ways to "clean up" after a Non-Steam Game has been removed, such as the ability to remove compatdatas, game artwork, and STL config files/other STL files for games which are no longer in the shortcuts.vdf. This will require parsing the shortcuts.vdf file, but now that I understand more about the format, this is now in the realm of possibility! Another neat feature would be a dialogue to view Non-Steam Games, similar to what DavidoTek is currently working on for ProtonUp-Qt, though without the ability to edit (as a result of Yad limitations, and also ProtonUp-Qt is doing is so we don't really need to I guess unless there's demand for it 🙂).

---

My year-long journey to understand these whacky Non-Steam AppIDs ends here for now. And very happily, I can say that this issue can now be closed with a happy resolution :-)

[trentondyck]

I'm excited to test/implement these updates downstream, A bit busy now but will take a look soon as I get some time. I'm doing some manual hacky (unreliable) stuff to modify the config.vdf adding the proton compatibility so I'd definitely be a customer for that feature as well.

[sonic2kk]

Took me a long time to figure it all out, if you run into issues it might be better to open a separate issue tracking problems. Just because this issue is a bit old and mainly tracking the feature request. But if you feel it's better here please comment and re-open for visibility (so I don't miss it 😅).

There are a couple of caveats which I guess I failed to mention, but very exotic filenames (I have some with blah.blah-blahblah.blah in the name) will not work, as this is also a limitation of the Set Game Art logic. So if you can use "clean" filenames that don't have a bunch of dots everywhere, it should work. Since this is being used downstream, it should be possible somewhere to rename the input files, or to at least parse them in some way :-)

---

As for the config.vdf changes, it should be feasible to do. Bash is... fun, when it comes to doing file operations like this, but it should be possible to echo an entry for the Non-Steam Game into that file, since we have the AppID now. I'd just need to make sure when implementing that it's as safe as possible, as messing up the config.vdf would be, well, a fucking disaster to be blunt and has the likelihood of destroying any set compatibility tools for all Steam games - Meaning into the thousands of games could be affected. So this feature would need a lot of thought and careful implementation, so no ETA on when I'll add it. But, hopefully it won't take another 7 months 😉

[trentondyck]

yes, of course. #905 added to continue later.