Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

flatdeb-steam documentation to build sniper is out of date #737

Open
KatyushaScarlet opened this issue Jan 20, 2025 · 13 comments · Fixed by #740
Open

flatdeb-steam documentation to build sniper is out of date #737

KatyushaScarlet opened this issue Jan 20, 2025 · 13 comments · Fixed by #740

Comments

@KatyushaScarlet
Copy link

Hi, I'm trying to build the sniper runtime with this script, but I got an error:

$ ./build-runtime.py \
--verbose \
---suite sniper \
--architecture amd64 \
--architecture i386 \
--archive ./archives/ \
--compression xz 

error: unrecognized arguments: ---suite sniper

Then I found that the args.suite only accept scout, scout_beta, heavy and heavy_beta.

I wonder if Ii'm using the right script or not.

Thanks!

@TTimo
Copy link
Collaborator

TTimo commented Jan 20, 2025

Hello,

Is there a specific reason you want to build a runtime yourself, vs using an official image? For more info about the Steam for Linux runtime, and how to obtain sdk and runtime builds, see https://gitlab.steamos.cloud/steamrt/steam-runtime-tools/-/blob/main/docs/slr-for-game-developers.md

@smcv we should probably cleanup those scripts, retire the chroot stuff and the old runtimes that we don't support anymore ..

@smcv
Copy link
Contributor

smcv commented Jan 21, 2025

It is technically possible to do your own builds of a sniper runtime, but I wouldn't recommend it. Because the container runtime is structured rather differently, they are done via a different set of scripts (https://gitlab.steamos.cloud/steamrt/flatdeb-steam, followed by populate-depot.py from the https://gitlab.steamos.cloud/steamrt/steam-runtime-tools repository).

We've generally been avoiding adding new tooling to this particular git repo, because its git history is inconveniently large: at some point in the past a developer committed binary .deb files to it, and now those are permanently part of the history and can't be removed. flatdeb-steam and steam-runtime-tools don't have that problem.

smcv added a commit to smcv/steam-runtime that referenced this issue Jan 27, 2025
Some of the scripts in this repository are still necessary when used
internally (in particular, we still use `build-runtime.py` to build
official scout releases), but third-party developers shouldn't normally
need to run them, and they are not the same scripts that we use to build
the container-based runtimes.

Resolves: ValveSoftware#737
Signed-off-by: Simon McVittie <smcv@collabora.com>
smcv added a commit to smcv/steam-runtime that referenced this issue Jan 27, 2025
Resolves: ValveSoftware#737
Signed-off-by: Simon McVittie <smcv@collabora.com>
@smcv
Copy link
Contributor

smcv commented Jan 27, 2025

we should probably cleanup those scripts, retire the chroot stuff and the old runtimes that we don't support anymore

I made a start on this in #740.

@smcv
Copy link
Contributor

smcv commented Jan 27, 2025

unrecognized arguments: ---suite sniper

FYI, the problem indicated by this error message wasn't actually that you were trying to build a sniper runtime with the wrong scripts. Instead, the problem is that it's spelled --suite, not ---suite.

(But if you'd spelled that option correctly, the result still wouldn't have been practically useful: there is no LD_LIBRARY_PATH-based runtime for sniper.)

TTimo pushed a commit that referenced this issue Jan 28, 2025
Some of the scripts in this repository are still necessary when used
internally (in particular, we still use `build-runtime.py` to build
official scout releases), but third-party developers shouldn't normally
need to run them, and they are not the same scripts that we use to build
the container-based runtimes.

Resolves: #737
Signed-off-by: Simon McVittie <smcv@collabora.com>
TTimo pushed a commit that referenced this issue Jan 28, 2025
Resolves: #737
Signed-off-by: Simon McVittie <smcv@collabora.com>
@KatyushaScarlet
Copy link
Author

KatyushaScarlet commented Feb 6, 2025

It is technically possible to do your own builds of a sniper runtime, but I wouldn't recommend it. Because the container runtime is structured rather differently, they are done via a different set of scripts (https://gitlab.steamos.cloud/steamrt/flatdeb-steam, followed by populate-depot.py from the https://gitlab.steamos.cloud/steamrt/steam-runtime-tools repository).

We've generally been avoiding adding new tooling to this particular git repo, because its git history is inconveniently large: at some point in the past a developer committed binary .deb files to it, and now those are permanently part of the history and can't be removed. flatdeb-steam and steam-runtime-tools don't have that problem.

@smcv
Thanks for the reply.
The reason I built the runtime myself is to customize it, and I want to explore the runtime technology used by Steam.
Now I'm trying to build the Sniper runtime but encountered some errors.
I followed the instructions here and here.
I am using Debian 12 and have followed the instructions to install all the required packages.

Here's my build command:

$ sudo subprojects/flatdeb/run.py --suite=sniper --arch=amd64,i386 base

Here's the part of the log.

2025/02/06 03:12:43 ==== run ====
2025/02/06 03:12:43 additional base packages | + install -d -m755 /scratch/root/run
2025/02/06 03:12:43 additional base packages | + echo [steamrt-customizations] 
2025/02/06 03:12:43 additional base packages | + /tmp/flatdeb.db9qrk6j/apt-install --debug --update --without-recommends /scratch/root /scratch/root/run/packages.yaml
2025/02/06 03:12:43 additional base packages | Hit:1 http://deb.debian.org/debian bullseye InRelease
2025/02/06 03:12:43 additional base packages | Get:2 http://security.debian.org/debian-security bullseye-security InRelease [27.2 kB]
2025/02/06 03:12:43 additional base packages | Ign:3 https://repo.steampowered.com/steamrt3/apt sniper InRelease
2025/02/06 03:12:43 additional base packages | Get:4 http://deb.debian.org/debian-debug bullseye-debug InRelease [37.6 kB]
2025/02/06 03:12:44 additional base packages | Get:5 http://security.debian.org/debian-security bullseye-security/non-free Sources [1352 B]
2025/02/06 03:12:44 additional base packages | Get:6 http://deb.debian.org/debian-debug bullseye-proposed-updates-debug InRelease [46.5 kB]
2025/02/06 03:12:44 additional base packages | Ign:7 https://repo.steampowered.com/steamrt3/apt sniper-runtime-overlay InRelease
2025/02/06 03:12:44 additional base packages | Get:8 http://deb.debian.org/debian bullseye/contrib Sources [43.2 kB]
2025/02/06 03:12:44 additional base packages | Get:9 http://deb.debian.org/debian bullseye/main Sources [8500 kB]
2025/02/06 03:12:44 additional base packages | Get:10 http://security.debian.org/debian-security bullseye-security/main Sources [232 kB]
2025/02/06 03:12:44 additional base packages | Ign:3 https://repo.steampowered.com/steamrt3/apt sniper InRelease
2025/02/06 03:12:45 additional base packages | Get:11 http://security.debian.org/debian-security bullseye-security/main amd64 Packages [341 kB]
2025/02/06 03:12:45 additional base packages | Ign:7 https://repo.steampowered.com/steamrt3/apt sniper-runtime-overlay InRelease
2025/02/06 03:12:45 additional base packages | Ign:3 https://repo.steampowered.com/steamrt3/apt sniper InRelease
2025/02/06 03:12:45 additional base packages | Get:12 http://security.debian.org/debian-security bullseye-security/main Translation-en [221 kB]
2025/02/06 03:12:46 additional base packages | Ign:7 https://repo.steampowered.com/steamrt3/apt sniper-runtime-overlay InRelease
2025/02/06 03:12:46 additional base packages | Get:13 http://security.debian.org/debian-security bullseye-security/non-free amd64 Packages [1164 B]
2025/02/06 03:12:46 additional base packages | Get:14 http://security.debian.org/debian-security bullseye-security/non-free Translation-en [1092 B]
2025/02/06 03:12:46 additional base packages | Err:3 https://repo.steampowered.com/steamrt3/apt sniper InRelease
2025/02/06 03:12:46 additional base packages |   Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown.  Could not handshake: Error in the certificate verification. [IP: 104.16.218.108 443]
2025/02/06 03:12:47 additional base packages | Err:7 https://repo.steampowered.com/steamrt3/apt sniper-runtime-overlay InRelease
2025/02/06 03:12:47 additional base packages |   Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown.  Could not handshake: Error in the certificate verification. [IP: 104.16.218.108 443]
2025/02/06 03:12:55 additional base packages | Get:15 http://deb.debian.org/debian bullseye/non-free Sources [81.0 kB]
2025/02/06 03:12:55 additional base packages | Get:16 http://deb.debian.org/debian bullseye/main Translation-en [6235 kB]
2025/02/06 03:12:56 additional base packages | Get:17 http://deb.debian.org/debian bullseye/contrib Translation-en [46.9 kB]
2025/02/06 03:12:56 additional base packages | Get:18 http://deb.debian.org/debian bullseye/non-free Translation-en [92.5 kB]
2025/02/06 03:12:56 additional base packages | Get:19 http://deb.debian.org/debian-debug bullseye-debug/main amd64 Packages [3317 kB]
2025/02/06 03:12:56 additional base packages | Get:20 http://deb.debian.org/debian-debug bullseye-debug/contrib amd64 Packages [33.9 kB]
2025/02/06 03:12:56 additional base packages | Get:21 http://deb.debian.org/debian-debug bullseye-debug/non-free amd64 Packages [16.4 kB]
2025/02/06 03:12:56 additional base packages | Get:22 http://deb.debian.org/debian-debug bullseye-proposed-updates-debug/main amd64 Packages [1032 B]
2025/02/06 03:12:56 additional base packages | Fetched 19.3 MB in 14s (1403 kB/s)
2025/02/06 03:12:57 additional base packages | Reading package lists...
2025/02/06 03:12:57 additional base packages | W: https://repo.steampowered.com/steamrt3/apt/dists/sniper/InRelease: No system certificates available. Try installing ca-certificates.
2025/02/06 03:12:57 additional base packages | W: https://repo.steampowered.com/steamrt3/apt/dists/sniper-runtime-overlay/InRelease: No system certificates available. Try installing ca-certificates.
2025/02/06 03:12:57 additional base packages | W: https://repo.steampowered.com/steamrt3/apt/dists/sniper/InRelease: No system certificates available. Try installing ca-certificates.
2025/02/06 03:12:57 additional base packages | W: https://repo.steampowered.com/steamrt3/apt/dists/sniper-runtime-overlay/InRelease: No system certificates available. Try installing ca-certificates.
2025/02/06 03:12:57 additional base packages | W: https://repo.steampowered.com/steamrt3/apt/dists/sniper/InRelease: No system certificates available. Try installing ca-certificates.
2025/02/06 03:12:57 additional base packages | W: https://repo.steampowered.com/steamrt3/apt/dists/sniper-runtime-overlay/InRelease: No system certificates available. Try installing ca-certificates.
2025/02/06 03:12:57 additional base packages | W: https://repo.steampowered.com/steamrt3/apt/dists/sniper/InRelease: No system certificates available. Try installing ca-certificates.
2025/02/06 03:12:57 additional base packages | W: https://repo.steampowered.com/steamrt3/apt/dists/sniper-runtime-overlay/InRelease: No system certificates available. Try installing ca-certificates.
2025/02/06 03:12:57 additional base packages | W: Failed to fetch https://repo.steampowered.com/steamrt3/apt/dists/sniper/InRelease  Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown.  Could not handshake: Error in the certificate verification. [IP: 104.16.218.108 443]
2025/02/06 03:12:57 additional base packages | W: Failed to fetch https://repo.steampowered.com/steamrt3/apt/dists/sniper-runtime-overlay/InRelease  Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown.  Could not handshake: Error in the certificate verification. [IP: 104.16.218.108 443]
2025/02/06 03:12:57 additional base packages | W: Some index files failed to download. They have been ignored, or old ones used instead.
2025/02/06 03:12:57 additional base packages | Reading package lists...
2025/02/06 03:12:57 additional base packages | Prio of /var/lib/dpkg/status 100
2025/02/06 03:12:57 additional base packages | Prio of /var/lib/apt/lists/security.debian.org_debian-security_dists_bullseye-security_non-free_i18n_Translation-en 100
2025/02/06 03:12:57 additional base packages | Prio of /var/lib/apt/lists/security.debian.org_debian-security_dists_bullseye-security_non-free_binary-amd64_Packages 500
2025/02/06 03:12:57 additional base packages | Prio of /var/lib/apt/lists/security.debian.org_debian-security_dists_bullseye-security_main_i18n_Translation-en 100
2025/02/06 03:12:57 additional base packages | Prio of /var/lib/apt/lists/security.debian.org_debian-security_dists_bullseye-security_main_binary-amd64_Packages 500
2025/02/06 03:12:57 additional base packages | Prio of /var/lib/apt/lists/deb.debian.org_debian-debug_dists_bullseye-proposed-updates-debug_main_binary-amd64_Packages 500
2025/02/06 03:12:57 additional base packages | Prio of /var/lib/apt/lists/deb.debian.org_debian-debug_dists_bullseye-debug_non-free_binary-amd64_Packages 500
2025/02/06 03:12:57 additional base packages | Prio of /var/lib/apt/lists/deb.debian.org_debian-debug_dists_bullseye-debug_contrib_binary-amd64_Packages 500
2025/02/06 03:12:57 additional base packages | Prio of /var/lib/apt/lists/deb.debian.org_debian-debug_dists_bullseye-debug_main_binary-amd64_Packages 500
2025/02/06 03:12:57 additional base packages | Prio of /var/lib/apt/lists/deb.debian.org_debian_dists_bullseye_non-free_i18n_Translation-en 100
2025/02/06 03:12:57 additional base packages | Prio of /var/lib/apt/lists/deb.debian.org_debian_dists_bullseye_non-free_binary-amd64_Packages 500
2025/02/06 03:12:57 additional base packages | Prio of /var/lib/apt/lists/deb.debian.org_debian_dists_bullseye_contrib_i18n_Translation-en 100
2025/02/06 03:12:57 additional base packages | Prio of /var/lib/apt/lists/deb.debian.org_debian_dists_bullseye_contrib_binary-amd64_Packages 500
2025/02/06 03:12:57 additional base packages | Prio of /var/lib/apt/lists/deb.debian.org_debian_dists_bullseye_main_i18n_Translation-en 100
2025/02/06 03:12:57 additional base packages | Prio of /var/lib/apt/lists/deb.debian.org_debian_dists_bullseye_main_binary-amd64_Packages 500
2025/02/06 03:12:57 additional base packages | Building dependency tree...
2025/02/06 03:12:57 additional base packages | E: Unable to locate package steamrt-customizations
2025/02/06 03:12:57 additional base packages | ERROR:flatdeb.apt-install:Command '['systemd-nspawn', '--directory=/scratch/root', '--as-pid2', '--pipe', '--tmpfs=/run/lock', '--register=no', 'env', 'DEBIAN_FRONTEND=noninteractive', 'apt-get', '-y', '-q', '-oAcquire::Retries=3', '-oDebug::pkgDepCache::AutoInstall=true', '-oDebug::pkgDepCache::Marker=true', '-oDebug::pkgPolicy=true', '-oDebug::pkgProblemResolver=true', '-oDebug::pkgProblemResolver::ShowScores=true', '--no-install-recommends', 'install', 'steamrt-customizations']' returned non-zero exit status 100.
2025/02/06 03:12:57 Action `run` failed at stage Run, error: exit status 1
ERROR:flatdeb:Command '['debos', '--artifactdir=/root/.cache/flatdeb', '-t', 'architecture:amd64', '-t', 'suite:bullseye', '-t', 'mirror:http://deb.debian.org/debian', '-t', 'ospack:base-sniper-amd64,i386.tar.gz.new', '-t', 'artifact_prefix:base-sniper-amd64,i386', '-t', 'foreignarchs:i386', '-t', 'mergedusr:after_sysroot', '-t', 'strip_source_version_suffix:\\+bsrt3[.~].*', '-t', 'additional_base_packages:[steamrt-customizations] ', '-t', 'keyring:suites/bullseye/overlay/etc/apt/trusted.gpg.d/flatdeb-build-0d6c9793-debian-archive-bullseye-stable.gpg', '-t', 'components:main,contrib,non-free', '/tmp/flatdeb.db9qrk6j/flatdeb.yaml']' returned non-zero exit status 1.

However, on my browser, the cert of repo.steampowered.com is trusted.

@smcv
Copy link
Contributor

smcv commented Feb 7, 2025

@kisak-valve, please could you reopen this and change the title to something like "flatdeb-steam documentation to build sniper out of date"?

@smcv
Copy link
Contributor

smcv commented Feb 7, 2025

Building runtimes with these tools, outside the CI infrastructure that produces official releases, is not something that is frequently done (you're perhaps the fourth or fifth person to have ever tried it!) so the readme probably doesn't have all the information that's needed.

$ sudo subprojects/flatdeb/run.py

You shouldn't need sudo for this, only an unprivileged user with access to /dev/kvm. The tools we're using for this (debos, fakemachine) use a virtual machine to get an environment where they can do setup that would normally require root.

Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown.

Do you have the ca-certificates package installed in your Debian 12 environment? That might need to be added to the list of required packages.

I think all the other errors you see are just a result of not being able to access the apt repository.

However, on my browser, the cert of repo.steampowered.com is trusted

Your browser has its own list of trusted CA certificates, which is independent of the one used by tools like apt, so it's entirely possible to get a situation where your browser works but apt doesn't.

@kisak-valve kisak-valve changed the title Can not build sniper suite with this script. flatdeb-steam documentation to build sniper is out of date Feb 7, 2025
@kisak-valve kisak-valve reopened this Feb 7, 2025
@KatyushaScarlet
Copy link
Author

KatyushaScarlet commented Feb 7, 2025

Replying to #737 (comment)

Yes, I did installed the ca-certificates package.
I also tried to not use sudo, but still got the same result at the end.

@smcv
Copy link
Contributor

smcv commented Feb 7, 2025

Hmm, I might see what's happening here. The final runtime includes ca-certificates, but the base stage doesn't, which means apt inside the chroot won't trust any https servers until after ca-certificates has been installed.

Try adding command-line options:

--replace-build-apt-source="steamrt=deb http://repo.steampowered.com/steamrt3/apt sniper main contrib non-free"

and

--replace-build-apt-source="steamrt-runtime-overlay=deb http://repo.steampowered.com/steamrt3/apt sniper-runtime-overlay main contrib non-free"

[edited: steamrt-runtime-overlay=, not sniper-runtime-overlay=]

(That's the same as the default, but with http instead of https. The result will still have secure integrity checking, using secure apt GPG signatures.)

Official builds aren't affected by this because we don't actually build from repo.steampowered.com. repo.steampowered.com only contains the newest stable and beta versions that have actually been released to the public, to avoid third-party developers picking up new dependencies that would make their games not work on the released version of the runtime. But when we make a new release of the runtime, it has to be built with newer packages than that, otherwise we wouldn't be able to update anything, so the new packages are taken from an internal apt repository.

@KatyushaScarlet
Copy link
Author

KatyushaScarlet commented Feb 8, 2025

Replying to #737 (comment)

Thanks!
These options resolved the cert issue, and the build seems to be successful.
However, I still encountered some errors at the end.
Additionally, I received many warnings like:

debootstrap | W: Unable to read /etc/apt/apt.conf.d/ - DirectoryExists (2: No such file or directory)

This is what I have in my directory:

$ ls -la  /etc/apt/apt.conf.d
total 16
drwxr-xr-x 2 root root 4096 Nov 30 18:40 .
drwxr-xr-x 8 root root 4096 Nov 30 18:40 ..
-rw-r--r-- 1 root root  399 May 25  2023 01autoremove
-rw-r--r-- 1 root root  182 Jan  9  2023 70debconf

Here is the end of the log.

2025/02/08 10:39:59 ==== Recipe done ====
INFO:flatdeb:Summarizing archive as mtree...
Traceback (most recent call last):
  File "/home/katyusha/build/flatdeb-steam/subprojects/flatdeb/run.py", line 2825, in <module>
    Builder().run_command_line()
  File "/home/katyusha/build/flatdeb-steam/subprojects/flatdeb/run.py", line 855, in run_command_line
    getattr(
  File "/home/katyusha/build/flatdeb-steam/subprojects/flatdeb/run.py", line 1179, in command_base
    self.generate_mtree(output, mtree)
  File "/home/katyusha/build/flatdeb-steam/subprojects/flatdeb/run.py", line 1188, in generate_mtree
    proc = subprocess.Popen(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 1024, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.11/subprocess.py", line 1901, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'bsdtar'

[Update]
After I install the libarchive-tools package, there's no issue now.
I think this package also should be documented.

@smcv
Copy link
Contributor

smcv commented Feb 8, 2025

Thanks, that sounds good. So I think there are two things we need to fix: the bootstrap should be done over http rather than https to avoid the ca-certificates dependency (this might just be a matter of configuration in flatdeb-steam, or we might need a small amount of new code in flatdeb to separate the build-time apt sources from the final apt sources first), and libarchive-tools needs adding to the list of required packages in the documentation (it's a new dependency, resulting from generating a mtree manifest as part of the build).

debootstrap | W: Unable to read /etc/apt/apt.conf.d/ - DirectoryExists (2: No such file or directory)

This warning is harmless: if we don't want to put any configuration in that directory (which we don't) then it doesn't matter that it doesn't exist. Possibly some part of flatdeb could avoid this warning by creating it as an empty directory.

@KatyushaScarlet
Copy link
Author

KatyushaScarlet commented Feb 9, 2025

Replying to #737 (comment)

Thank you for your reply!
But where can I find the build result?
After the script runs successfully, I can't find any tarball or other build output in the directory.
I followed these steps:

$ subprojects/flatdeb/run.py --suite=sniper --arch=amd64,i386 \
--replace-build-apt-source="steamrt=deb http://repo.steampowered.com/steamrt3/apt sniper main contrib non-free" \
--replace-build-apt-source="steamrt-runtime-overlay=deb http://repo.steampowered.com/steamrt3/apt sniper-runtime-overlay main contrib non-free" \
base

$ subprojects/flatdeb/run.py --suite=sniper --arch=amd64,i386 \
--replace-build-apt-source="steamrt=deb http://repo.steampowered.com/steamrt3/apt sniper main contrib non-free" \
--replace-build-apt-source="steamrt-runtime-overlay=deb http://repo.steampowered.com/steamrt3/apt sniper-runtime-overlay main contrib non-free" \
--no-collect-source-code \
--no-debug-symbols \
--no-generate-source-tarball \
--no-ostree-commit \
runtimes \
runtimes/sniper.yaml

Thanks!

[Update]
I found some tarballs in the ~/.cache/flatdeb directory.
I'll keep trying to build out the full sniper runtime as steam delivers.

@smcv
Copy link
Contributor

smcv commented Feb 18, 2025

But where can I find the build result?

I found some tarballs in the ~/.cache/flatdeb directory

Yes, that's the default location for output unless overridden by flatdeb/run.py --build-area=....

I'll keep trying to build out the full sniper runtime as steam delivers.

The official depot published via Steampipe consists of the contents of the archive SteamLinuxRuntime_sniper.tar.xz from https://repo.steampowered.com/steamrt3/images/latest-container-runtime-depot/ or similar, which is the output of populate-depot.py from the https://gitlab.steamos.cloud/steamrt/steam-runtime-tools repository, packed into a tarball.

The CI that runs flatdeb and populate-depot.py is not visible to the public, because it acts on unreleased versions of the packages that go into the container, so there is no obvious public location I can point you to that would describe the options that are passed to populate-depot.py for official builds; you'll likely need to experiment a bit.

As I'm sure you've realized by now, none of this is something that end users or game developers would normally need to do, and the official releases published by Valve are the only thing that we can provide support for.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants