Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

segmentation fault in Kubernetes #27

Open
alexstaeding opened this issue Nov 30, 2023 · 21 comments
Open

segmentation fault in Kubernetes #27

alexstaeding opened this issue Nov 30, 2023 · 21 comments

Comments

@alexstaeding
Copy link

This is the manifest I am using: https://github.com/alexstaeding/homelab/blob/d0412c55d9212f60ce7aac4e7a6c15030110561e/kubernetes/ttt/deploy.yaml

The container seems to work according to logs but terminates before starting the server. These are the last log messages before the container restarts:

LinuxGSM Stats
=================================
Assist LinuxGSM development by sending anonymous stats to developers.
More info: https://docs.linuxgsm.com/configuration/linuxgsm-stats
The following info will be sent:
* game server
* distro
* game server resource usage
* server hardware info
Information! auto-install leaves stats off by default. Stats can be enabled in common.cfg
fetching GitHub install_complete.sh...OK

=================================
Install Complete!

To start server type:
./gmodserver start

fetching GitHub core_exit.sh...OK
[entrypoint.sh]update / installation done!
[prepareServer.sh]check configurations
[initConfig.sh]Request for replacing hostname to "My Server", source is found 1 times
[initConfig.sh]Request for replacing sv_password to "...", source is found 1 times
@jusito
Copy link
Owner

jusito commented Nov 30, 2023

Heyho, the values are replaced via:

sed -Ei "/${source}.*/d" "${CFG_PATH}"

Currently the value is not escaped so I would expect your value isn't sed safe? E.g. contains /, \, [, ], &, *.
Its also an issue that here is extended regex used, Iam not sure if I can remove such issue completely.
Well I could check the string to contain only safe characters and if not at least warn and throw an error.

@alexstaeding
Copy link
Author

Hi, I did that replacement to ... myself to not leak the password. The actual password is alphanumeric, e.g: sEGb7Rx2mHWTgv94k5Krpt.

Its weird that there are no logs after the replacement, though.

@jusito
Copy link
Owner

jusito commented Nov 30, 2023

You can find out what happens by setting env var DEBUGGING to true which will add xtrace, so every cmd will be printed before execution.
I always set "exit on cmd fail", "unset variables are an error" and "if a single cmd in a pipeline fails, fail the pipeline".
So I suspected sed fails silently.

@alexstaeding
Copy link
Author

I guess I will just burn the password now. This is the container log with debugging enabled:

[initConfig.sh]Request for replacing hostname to "My Server", source is found 1 times
+ sed -Ei '/hostname.*/d' /home/steam/serverfiles/garrysmod/cfg/gmodserver.cfg
+ echo 'hostname "My Server"'
+ '[' -n m9gA3qDZSCPx6ue4vs8HQnMrWU57VkKpbhJTtaydFX ']'
+ configReplace sv_password m9gA3qDZSCPx6ue4vs8HQnMrWU57VkKpbhJTtaydFX
+ source=sv_password
+ target='"m9gA3qDZSCPx6ue4vs8HQnMrWU57VkKpbhJTtaydFX"'
++ grep -Poc '(sv_password).+' /home/steam/serverfiles/garrysmod/cfg/gmodserver.cfg
+ count=1
+ echo '[initConfig.sh]Request for replacing sv_password to "m9gA3qDZSCPx6ue4vs8HQnMrWU57VkKpbhJTtaydFX", source is found 1 times'
+ '[' 1 == 1 ']'
[initConfig.sh]Request for replacing sv_password to "m9gA3qDZSCPx6ue4vs8HQnMrWU57VkKpbhJTtaydFX", source is found 1 times
+ sed -Ei '/sv_password.*/d' /home/steam/serverfiles/garrysmod/cfg/gmodserver.cfg
+ echo 'sv_password "m9gA3qDZSCPx6ue4vs8HQnMrWU57VkKpbhJTtaydFX"'
+ '[' -n zkZ6SH7mWxAEt54pTwehDy8BGLjMsYcVbUKPvXu2Ff ']'
+ configReplace rcon_password zkZ6SH7mWxAEt54pTwehDy8BGLjMsYcVbUKPvXu2Ff
+ source=rcon_password
+ target='"zkZ6SH7mWxAEt54pTwehDy8BGLjMsYcVbUKPvXu2Ff"'
++ grep -Poc '(rcon_password).+' /home/steam/serverfiles/garrysmod/cfg/gmodserver.cfg
+ count=0

It immediately stops after that last command. The count=0 seems suspicious.

@alexstaeding
Copy link
Author

It seems to work if I remove SERVER_RCON_PASSWORD from the environment

@jusito
Copy link
Owner

jusito commented Nov 30, 2023

updated the dev build first, e7aecfe
Please give me feedback if that resolves your issue.

Edit: "...-develop" should work as tag

@jusito
Copy link
Owner

jusito commented Nov 30, 2023

docker hub needs probably around a few minutes to build it.

@alexstaeding
Copy link
Author

Thanks for the quick fix, I will look at that shortly. However, I may have spoken too soon about it working - it "starts" the server but I am not able to connect. I ruled out kubernetes/loadbalancing issues by trying the same setup with an nginx helloworld.

This is the end of the log:

...
 Update state (0x61) downloading, progress: 99.14 (2277460869 / 2297127618)
 Update state (0x81) verifying update, progress: 25.06 (575626259 / 2297127618)
 Update state (0x81) verifying update, progress: 67.63 (1553447477 / 2297127618)
Success! App '232330' fully installed.
Server is running, waiting for SIGTERM
fetching GitHub command_start.sh...OK
fetching GitHub check_root.sh...OK
fetching GitHub check_system_dir.sh...OK
fetching GitHub check_executable.sh...OK
fetching GitHub check_logs.sh...OK
[ INFO ] Starting gmodserver: Checking for log files: Creating log files
installing log dir: /home/steam/log...OK
installing LinuxGSM log dir: /home/steam/log/script...OK
creating LinuxGSM log: /home/steam/log/script/gmodserver-script.log...OK
installing console log dir: /home/steam/log/console...OK
creating console log: /home/steam/log/console/gmodserver-console.log...OK
creating symlink to game log dir: /home/steam/log/server -> /home/steam/serverfiles/garrysmod/logs...OK
creating symlink to steam log dir: /home/steam/log/steam -> /home/steam/.steam/steam/logs...OK
fetching GitHub check_config.sh...OK
fetching GitHub check_status.sh...OK
fetching GitHub fix_steamcmd.sh...OK
[  OK  ] Starting gmodserver: Applying steamclient.so sdk64 hardlink fix: Garry's Mod
[  OK  ] Starting gmodserver: Applying steamclient.so sdk32 link fix: Garry's Mod
fetching GitHub core_logs.sh...OK
[  OK  ] Starting gmodserver: My Server

Here is the TTT load balancer configuration:

apiVersion: v1
kind: Service
metadata:
  name: server-0
  namespace: ttt
  annotations:
    metallb.universe.tf/address-pool: address-pool
    metallb.universe.tf/loadBalancerIPs: 10.0.10.2
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  selector:
    statefulset.kubernetes.io/pod-name: server-0
  ports:
    - name: game-tcp
      protocol: TCP
      port: 27015
      targetPort: 27015
    - name: game-udp
      protocol: UDP
      port: 27015
      targetPort: 27015

A client on the local network is not able to join:
image

I thought this may be due to a load balancer issue, so I tried with a nginx helloworld.

The stateful set (to replicate ttt setup):

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nginx
  namespace: ttt
spec:
  serviceName: ttt-nginx
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: ttt-nxinx
          image: nginxdemos/hello
          ports:
            - containerPort: 80
              protocol: TCP

The load balancer, copied and slightly adjusted from ttt:

apiVersion: v1
kind: Service
metadata:
  name: nginx-0
  namespace: ttt
  annotations:
    metallb.universe.tf/address-pool: address-pool
    metallb.universe.tf/loadBalancerIPs: 10.0.10.3
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  selector:
    statefulset.kubernetes.io/pod-name: nginx-0
  ports:
    - name: web-tcp
      protocol: TCP
      port: 80
      targetPort: 80

Accessing 10.0.10.3 in a web browser works.

@alexstaeding
Copy link
Author

I just had a quick look at the gmodserver-script log, this doesn't look right:

steam@server-0:~/serverfiles/garrysmod/logs$ cat /home/steam/log/script/gmodserver-script.log
Nov 30 14:42:16.517 gmodserver: START: PASS: Started My Server
Nov 30 14:42:17.027 gmodserver: START: PASS: Started My Server
Nov 30 14:42:17.057 gmodserver: MONITOR: INFO: LinuxGSM version: v23.5.3
Nov 30 14:42:17.059 gmodserver: MONITOR: PASS: core_exit.sh exiting with code: 0
Nov 30 14:45:03.157 gmodserver: MONITOR: INFO: Checking session: CHECKING
Nov 30 14:45:04.179 gmodserver: MONITOR: PASS: Checking session: OK
Nov 30 14:45:04.185 gmodserver: MONITOR: INFO: gamedig is not installed
Nov 30 14:45:04.188 gmodserver: MONITOR: INFO: https://docs.linuxgsm.com/requirements/gamedig
Nov 30 14:45:05.195 gmodserver: MONITOR: INFO: Querying port: gsquery: 127.0.0.1:27015 : 1 : QUERYING
Nov 30 14:45:06.210 gmodserver: MONITOR: INFO: Querying port: gsquery: 0.0.0.0:27015 : 1 : DELAY
Nov 30 14:45:06.214 gmodserver: MONITOR: INFO: Query bypassed: gmodserver started less than 5 minutes ago
Nov 30 14:45:06.222 gmodserver: MONITOR: INFO: Server started: Thu Nov 30 14:42:16 CET 2023
Nov 30 14:45:06.227 gmodserver: MONITOR: INFO: Current time: Thu Nov 30 14:45:06 CET 2023
Nov 30 14:45:06.238 gmodserver: MONITOR: INFO: LinuxGSM version: v23.5.3
Nov 30 14:45:06.241 gmodserver: MONITOR: PASS: core_exit.sh exiting with code: 0
Nov 30 14:50:02.647 gmodserver: MONITOR: INFO: Checking session: CHECKING
Nov 30 14:50:03.670 gmodserver: MONITOR: PASS: Checking session: OK
Nov 30 14:50:03.676 gmodserver: MONITOR: INFO: gamedig is not installed
Nov 30 14:50:03.678 gmodserver: MONITOR: INFO: https://docs.linuxgsm.com/requirements/gamedig
Nov 30 14:50:04.687 gmodserver: MONITOR: INFO: Querying port: gsquery: 127.0.0.1:27015 : 1 : QUERYING
Nov 30 14:50:05.732 gmodserver: MONITOR: WARN: Querying port: gsquery: 127.0.0.1:27015 : 1 : FAIL
Nov 30 14:50:06.740 gmodserver: MONITOR: INFO: Querying port: gsquery: 10.42.2.142:27015 : 1 : QUERYING
Nov 30 14:50:07.784 gmodserver: MONITOR: WARN: Querying port: gsquery: 10.42.2.142:27015 : 1 : FAIL

@alexstaeding
Copy link
Author

Oh, and the console log shows a segmentation fault 🙃

https://gist.github.com/alexstaeding/d5952cb1cf6bf0a827f2b827eed7b86c

@jusito
Copy link
Owner

jusito commented Nov 30, 2023

Console log is a good idea because after [ OK ] Starting gmodserver: My Server the server downloads all mods so this could take a while until its there. Seqfault, did you try to start without any mod and with a fresh installation on the same node?

@alexstaeding
Copy link
Author

alexstaeding commented Nov 30, 2023

I just wiped everything (sts, pvc) and get the same segfault. This is a clean installation without mods.
I also noticed this earlier in the main container log, I don't know if it's relevant:

[----] Verifying installation...
Steam Console Client (c) Valve Corporation - version 1701290101
-- type 'quit' to exit --
Loading Steam API...dlmopen steamservice.so failed: steamservice.so: cannot open shared object file: No such file or directory
OK

What else can I look at?

@jusito
Copy link
Owner

jusito commented Nov 30, 2023

From a fresh start I have the same msg but server is working. Maybe important side note, gmod is very old e.g. it doesn't like if configured port and port from outside is not matching. So maybe there are issue with virtual networking but as I experienced it it didn't crashed it just wasnt reachable.

Maybe lets test this config:

volumes:
  TTTDebug:

services:
  debug:
    build: .
    ports:
      - 27016:27016/tcp
      - 27016:27016/udp
    environment:
      INSTALL_CSS: "true"
      SERVER_PORT: 27016
      SERVER_NAME: "Example Name"
      SERVER_PASSWORD: securePassword
      SERVER_VOICE_ENABLE: 0
      SERVER_MAX_PLAYERS: 20
      SERVER_RCON_PASSWORD: securePassword
    volumes:
      - TTTDebug:/home/steam/serverfiles
[entrypoint.sh]starting entrypoint.sh
[entrypoint.sh]installing...
fetching GitHub serverlist.csv...OK�[2K
Installed Garrys Mod server as gmodserver

Remember to check server ports
./gmodserver details

fetching GitHub core_modules.sh...OK�[2K
fetching GitHub core_legacy.sh...OK�[2K
fetching GitHub core_messages.sh...OK�[2K
fetching GitHub core_dl.sh...OK�[2K
fetching GitHub core_trap.sh...OK
fetching GitHub _default.cfg...OK
copying _default.cfg...OK
fetching GitHub check_ip.sh...OK
fetching GitHub info_game.sh...OK
fetching GitHub common.cfg...OK
fetching GitHub secrets-common.cfg...OK
fetching GitHub gmodserver.cfg...OK
fetching GitHub secrets-gmodserver.cfg...OK
fetching GitHub linuxgsm.sh...OK
fetching GitHub core_getopt.sh...�[32mOK�[0m
fetching GitHub command_install.sh...�[32mOK�[0m
fetching GitHub check.sh...�[32mOK�[0m
fetching GitHub check_version.sh...�[32mOK�[0m
fetching GitHub check_tmuxception.sh...�[32mOK�[0m
fetching GitHub check_permissions.sh...�[32mOK�[0m
fetching GitHub check_glibc.sh...�[32mOK�[0m
fetching GitHub info_distro.sh...�[32mOK�[0m
fetching GitHub check_system_requirements.sh...�[32mOK�[0m
fetching GitHub install_header.sh...�[32mOK�[0m
�[H�[2J�[3J
                                mdMMMMbm
                              mMMMMMMMMMMm
                              mMMMMMMMMMMMMm
                             mMMMMMMMMMMMMMMm
                             hMMMV^VMMV^VMMMh
                             MMMMM  MM  MMMMM
                             hMMs   vv   sMMh
                            hMMM:        :MMMh
                          .hMMMh          hMMMh.
                         -dMMMh     �[37m__�[0m     hMMMd-
                        :mMMMs      �[37m||�[0m      sMMMm:
                       :MMMM+       �[37m||�[0m �[31m_�[0m     +NMMN:
                      .mMMM+     �[37m========�[0m     +MMMm.
                      yMMMy   �[90m##############�[0m   yMMMy
                      mMMM:   �[90m##############�[0m   :MMMm
                      mMM   �[93mnn�[0m   �[93mnn�[0m    �[93mnn�[0m   �[93mnn�[0m   MMm
                      o   �[93mnNNNNNNNn�[0m    �[93mnNNNNNNNn�[0m   o
                         �[93mnNNNNNNNNNn�[0m  �[93mnNNNNNNNNNn�[0m
                        �[93mnNNNNNNNNNNN�[0m  �[93mNNNNNNNNNNNn�[0m
                         �[93m+NNNNNNNNN:�[0m  �[93m:NNNNNNNNN+�[0m
                           �[93mnNNNNNNN�[0m /\ �[93mNNNNNNNn�[0m
                             �[93mnnnnn�[0m  db  �[93mnnnnn�[0m

�[93m888�[0m      �[93md8b�[0m                             �[0m.d8888b.   .d8888b.  888b     d888
�[93m888      Y8P                            �[0md88P  Y88b d88P  Y88b 8888b   d8888
�[93m888�[0m                                     �[0m888�[0m    888 Y88b.      88888b.d88888
�[93m888�[0m      �[93m888�[0m �[93m88888b.�[0m  �[93m888�[0m  �[93m888�[0m �[93m888�[0m  �[93m888�[0m 888          Y888b.   888Y88888P888
�[93m888�[0m      �[93m888�[0m �[93m888�[0m  �[93m88b�[0m �[93m888�[0m  �[93m888�[0m  �[93mY8bd8P�[0m  888  88888      Y88b. 888 Y888P 888
�[93m888�[0m      �[93m888�[0m �[93m888�[0m  �[93m888�[0m �[93m888�[0m  �[93m888�[0m   �[93mX88K�[0m   888    888        888 888  Y8P  888
�[93m888�[0m      �[93m888�[0m �[93m888�[0m  �[93m888�[0m �[93mY88b�[0m �[93m88Y�[0m �[93m.d8pq8b.�[0m Y88b  d88P Y88b  d88P 888   *   888
�[93mLinuxGSM�[0m �[93m888�[0m �[93m888�[0m  �[93m888�[0m  �[93mY8888Y�[0m  �[93m888�[0m  �[93m888�[0m   Y2012P88   Y8888P   888       888

=================================�[0m
�[93mLinux�[0mGSM_�[0m
by Daniel Gibbs�[0m
�[94mVersion:�[0m v23.6.2
�[94mGame:�[0m Garry's Mod
�[94mWebsite:�[0m https://linuxgsm.com
�[94mContributors:�[0m https://linuxgsm.com/contrib
�[94mSponsor:�[0m https://linuxgsm.com/sponsor
=================================�[0m
fetching GitHub install_server_dir.sh...�[32mOK�[0m

�[93mServer Directory�[0m
=================================�[0m
�[93mWarning!�[0m A server is already installed here.
/home/steam
fetching GitHub install_logs.sh...�[32mOK�[0m

�[93mCreating Log Directories�[0m
=================================�[0m
installing log dir: /home/steam/log...�[32mOK�[0m
installing LinuxGSM log dir: /home/steam/log/script...�[32mOK�[0m
creating LinuxGSM log: /home/steam/log/script/gmodserver-script.log...�[32mOK�[0m
installing console log dir: /home/steam/log/console...�[32mOK�[0m
creating console log: /home/steam/log/console/gmodserver-console.log...�[32mOK�[0m
installing game log dir: /home/steam/serverfiles/garrysmod/logs...�[32mOK�[0m
creating symlink to game log dir: /home/steam/log/server -> /home/steam/serverfiles/garrysmod/logs...�[32mOK�[0m
fetching GitHub check_deps.sh...�[32mOK�[0m

�[93mChecking Garry's Mod Dependencies�[0m
=================================�[0m
checking GitHub debian-10.csv...�[32mOK�[0m�[2K
fetching GitHub debian-10.csv...�[32mOK�[0m
�[32mbc�[0m
�[32mbinutils�[0m
�[32mbsdmainutils�[0m
�[32mbzip2�[0m
�[32mca-certificates�[0m
�[32mcpio�[0m
�[32mcurl�[0m
�[32mdistro-info�[0m
�[32mfile�[0m
�[32mgzip�[0m
�[32mhostname�[0m
�[32mjq�[0m
�[32mlib32gcc1�[0m
�[32mlib32stdc++6�[0m
�[32mlibsdl2-2.0-0:i386�[0m
�[32mlibtinfo5:i386�[0m
�[32mnetcat�[0m
�[32mpython3�[0m
�[32mtar�[0m
�[32mtmux�[0m
�[32munzip�[0m
�[32mutil-linux�[0m
�[32muuid-runtime�[0m
�[32mwget�[0m
�[32mxz-utils�[0m
�[36mInformation!�[0m Required dependencies already installed.
fetching GitHub install_steamcmd.sh...�[32mOK�[0m

�[93mInstalling SteamCMD�[0m
=================================�[0m
fetching GitHub check_steamcmd.sh...�[32mOK�[0m
fetching GitHub core_steamcmd.sh...�[32mOK�[0m
downloading steamcmd_linux.tar.gz...�[1K
######################################################################## 100.0%
�[32mOK�[0m
extracting steamcmd_linux.tar.gz...�[32mOK�[0m
fetching GitHub install_server_files.sh...�[32mOK�[0m

�[93mInstalling Garry's Mod Server�[0m
=================================�[0m
fetching GitHub update_steamcmd.sh...�[32mOK�[0m

�[K[ .... ] Installing gmodserver: SteamCMD
�[K[�[92m START �[0m] Installing gmodserver: SteamCMD
Redirecting stderr to '/home/steam/.local/share/Steam/logs/stderr.txt'
ILocalize::AddFile() failed to load file "public/steambootstrapper_english.txt".
[  0%] Checking for available update...
[----] Downloading update (0 of 59659 KB)...
[  0%] Downloading update (0 of 59659 KB)...
[  0%] Downloading update (3493 of 59659 KB)...
[  5%] Downloading update (7206 of 59659 KB)...
[ 12%] Downloading update (9997 of 59659 KB)...
[ 16%] Downloading update (12543 of 59659 KB)...
[ 21%] Downloading update (15088 of 59659 KB)...
[ 25%] Downloading update (17886 of 59659 KB)...
[ 29%] Downloading update (20419 of 59659 KB)...
[ 34%] Downloading update (22951 of 59659 KB)...
[ 38%] Downloading update (25492 of 59659 KB)...
[ 42%] Downloading update (28286 of 59659 KB)...
[ 47%] Downloading update (30815 of 59659 KB)...
[ 51%] Downloading update (33367 of 59659 KB)...
[ 55%] Downloading update (35905 of 59659 KB)...
[ 60%] Downloading update (38715 of 59659 KB)...
[ 64%] Downloading update (41185 of 59659 KB)...
[ 69%] Downloading update (42459 of 59659 KB)...
[ 71%] Downloading update (43860 of 59659 KB)...
[ 73%] Downloading update (45133 of 59659 KB)...
[ 75%] Downloading update (46407 of 59659 KB)...
[ 77%] Downloading update (47807 of 59659 KB)...
[ 80%] Downloading update (49086 of 59659 KB)...
[ 82%] Downloading update (50358 of 59659 KB)...
[ 84%] Downloading update (51757 of 59659 KB)...
[ 86%] Downloading update (53030 of 59659 KB)...
[ 88%] Downloading update (54308 of 59659 KB)...
[ 91%] Downloading update (55709 of 59659 KB)...
[ 93%] Downloading update (56988 of 59659 KB)...
[ 95%] Downloading update (58262 of 59659 KB)...
[ 97%] Downloading update (59659 of 59659 KB)...
[100%] Downloading update (59659 of 59659 KB)...
[100%] Download Complete.
[----] Applying update...
[----] Extracting package...
[----] Installing update...
[----] Cleaning up...
[----] Update complete, launching...
WARNING: setlocale('en_US.UTF-8') failed, using locale: 'C'. International characters may not work.
Redirecting stderr to '/home/steam/.local/share/Steam/logs/stderr.txt'
Logging directory: '/home/steam/.local/share/Steam/logs'
/tmp/dumps insufficient permissions - delete and recreate
[  0%] Checking for available updates...
[----] Verifying installation...
[  0%] Downloading update...
[  0%] Checking for available updates...
[----] Download complete.
[----] Extracting package...
[----] Installing update...
[----] Cleaning up...
[----] Update complete, launching Steamcmd...
WARNING: setlocale('en_US.UTF-8') failed, using locale: 'C'. International characters may not work.
Redirecting stderr to '/home/steam/.local/share/Steam/logs/stderr.txt'
Logging directory: '/home/steam/.local/share/Steam/logs'
[  0%] Checking for available updates...
[----] Verifying installation...
Steam Console Client (c) Valve Corporation - version 1701290101
-- type 'quit' to exit --
Loading Steam API...dlmopen steamservice.so failed: steamservice.so: cannot open shared object file: No such file or directory
OK

Connecting anonymously to Steam Public...OK
Waiting for client config...OK
Waiting for user info...OK
 Update state (0x3) reconfiguring, progress: 0.00 (0 / 0)
 Update state (0x61) downloading, progress: 0.13 (5242880 / 4068729591)
 Update state (0x61) downloading, progress: 1.66 (67573552 / 4068729591)
 Update state (0x61) downloading, progress: 2.13 (86727779 / 4068729591)
 Update state (0x61) downloading, progress: 3.75 (152621066 / 4068729591)
 Update state (0x61) downloading, progress: 5.34 (217444368 / 4068729591)
 Update state (0x61) downloading, progress: 6.54 (266179931 / 4068729591)
 Update state (0x61) downloading, progress: 8.95 (364315851 / 4068729591)
 Update state (0x61) downloading, progress: 13.29 (540782495 / 4068729591)
 Update state (0x61) downloading, progress: 16.67 (678259329 / 4068729591)
 Update state (0x61) downloading, progress: 20.20 (821765891 / 4068729591)
 Update state (0x61) downloading, progress: 24.40 (992731669 / 4068729591)
 Update state (0x61) downloading, progress: 31.69 (1289366112 / 4068729591)
 Update state (0x61) downloading, progress: 37.74 (1535341287 / 4068729591)
 Update state (0x61) downloading, progress: 42.78 (1740796247 / 4068729591)
 Update state (0x61) downloading, progress: 47.68 (1940118659 / 4068729591)
 Update state (0x61) downloading, progress: 52.96 (2154613333 / 4068729591)
 Update state (0x61) downloading, progress: 56.05 (2280387496 / 4068729591)
 Update state (0x61) downloading, progress: 59.33 (2414111008 / 4068729591)
 Update state (0x61) downloading, progress: 64.25 (2614065193 / 4068729591)
 Update state (0x61) downloading, progress: 69.06 (2809714002 / 4068729591)
 Update state (0x61) downloading, progress: 74.27 (3021993756 / 4068729591)
 Update state (0x61) downloading, progress: 80.88 (3290734920 / 4068729591)
 Update state (0x61) downloading, progress: 87.36 (3554391491 / 4068729591)
 Update state (0x61) downloading, progress: 92.53 (3764613133 / 4068729591)
 Update state (0x61) downloading, progress: 95.03 (3866562735 / 4068729591)
 Update state (0x61) downloading, progress: 98.25 (3997395887 / 4068729591)
 Update state (0x81) verifying update, progress: 0.59 (24117248 / 4068729591)
 Update state (0x81) verifying update, progress: 15.05 (612432272 / 4068729591)
 Update state (0x81) verifying update, progress: 29.29 (1191632620 / 4068729591)
 Update state (0x81) verifying update, progress: 43.58 (1773126800 / 4068729591)
 Update state (0x81) verifying update, progress: 58.15 (2365772685 / 4068729591)
 Update state (0x81) verifying update, progress: 72.47 (2948548527 / 4068729591)
 Update state (0x81) verifying update, progress: 86.88 (3534890340 / 4068729591)
Success! App '4020' fully installed.
�[32mComplete!�[0m Installing gmodserver: SteamCMD
fetching GitHub install_config.sh...�[32mOK�[0m

�[93mDownloading Garry's Mod Configs�[0m
=================================�[0m
Downloading default configs from:

https://github.com/GameServerManagers/Game-Server-Configs�[0m

fetching GitHub server.cfg...�[32mOK�[0m
copying server.cfg config file.
'/home/steam/lgsm/config-default/config-game/server.cfg' -> '/home/steam/serverfiles/garrysmod/cfg/gmodserver.cfg'
changing hostname.
changing rcon/admin password.

�[93mConfig Locations�[0m
=================================�[0m
Game Server Config File: /home/steam/serverfiles/garrysmod/cfg/gmodserver.cfg
LinuxGSM Config: /home/steam/lgsm/config-lgsm/gmodserver
Documentation: https://docs.linuxgsm.com/configuration/game-server-config
fetching GitHub install_gslt.sh...�[32mOK�[0m

�[93mGame Server Login Token�[0m
=================================�[0m
GSLT is an optional feature for Garry's Mod server
Get more info and a token here:
https://docs.linuxgsm.com/steamcmd/gslt

The GSLT can be changed by editing /home/steam/lgsm/config-lgsm/gmodserver/gmodserver.cfg.

fetching GitHub fix.sh...�[32mOK�[0m
fetching GitHub install_stats.sh...�[32mOK�[0m

�[93mLinuxGSM Stats�[0m
=================================�[0m
Assist LinuxGSM development by sending anonymous stats to developers.
Collected data is publicly available: https://linuxgsm.com/data/usage�[0m
More info: https://docs.linuxgsm.com/configuration/linuxgsm-stats�[0m
The following info will be sent: 
* game server
* distro
* game server resource usage
* server hardware info
�[36mInformation!�[0m auto-install leaves stats off by default. Stats can be enabled in common.cfg
fetching GitHub install_complete.sh...�[32mOK�[0m

=================================�[0m
�[32mInstall Complete!�[0m

To start server type:
./gmodserver start

fetching GitHub core_exit.sh...�[32mOK�[0m
[entrypoint.sh]update / installation done!
[prepareServer.sh]check configurations
[initConfig.sh]Warning your configured value for hostname doesn't match regex ^[a-zA-Z0-9_-]+$. Special symbols could crash this script because they are not sanitized.
[initConfig.sh]Request for replacing hostname to "Example Name", source is found 1 times
[initConfig.sh]Warning your configured value for sv_password doesn't match regex ^[a-zA-Z0-9_-]+$. Special symbols could crash this script because they are not sanitized.
[initConfig.sh]Request for replacing sv_password to "securePassword", source is found 1 times
[initConfig.sh]Warning your configured value for rcon_password doesn't match regex ^[a-zA-Z0-9_-]+$. Special symbols could crash this script because they are not sanitized.
[initConfig.sh]Request for replacing rcon_password to "securePassword", source is found 1 times
[initConfig.sh]Warning your configured value for sv_voiceenable doesn't match regex ^[a-zA-Z0-9_-]+$. Special symbols could crash this script because they are not sanitized.
[initConfig.sh]Request for replacing sv_voiceenable to "0", source is found 1 times
[prepareServer.sh]force workshop download
[forceWorkshopDownload.sh]No auto workshop download
[prepareServer.sh]install & mount gamefiles
[installAndMountAddons.sh]installing & mounting css
WARNING: setlocale('en_US.UTF-8') failed, using locale: 'C'. International characters may not work.
Redirecting stderr to '/home/steam/.local/share/Steam/logs/stderr.txt'
Logging directory: '/home/steam/.local/share/Steam/logs'
[  0%] Checking for available updates...
[----] Verifying installation...
Steam Console Client (c) Valve Corporation - version 1701290101
-- type 'quit' to exit --
Loading Steam API...dlmopen steamservice.so failed: steamservice.so: cannot open shared object file: No such file or directory
OK

Connecting anonymously to Steam Public...OK
Waiting for client config...OK
Waiting for user info...OK
 Update state (0x3) reconfiguring, progress: 0.00 (0 / 0)
 Update state (0x3) reconfiguring, progress: 0.00 (0 / 0)
 Update state (0x61) downloading, progress: 1.60 (36819853 / 2297127618)
 Update state (0x61) downloading, progress: 4.92 (112919015 / 2297127618)
 Update state (0x61) downloading, progress: 8.33 (191333837 / 2297127618)
 Update state (0x61) downloading, progress: 12.03 (276268493 / 2297127618)
 Update state (0x61) downloading, progress: 15.59 (358133858 / 2297127618)
 Update state (0x61) downloading, progress: 19.91 (457349752 / 2297127618)
 Update state (0x61) downloading, progress: 27.17 (624135263 / 2297127618)
 Update state (0x61) downloading, progress: 38.47 (883695887 / 2297127618)
 Update state (0x61) downloading, progress: 51.94 (1193020900 / 2297127618)
 Update state (0x61) downloading, progress: 66.31 (1523190791 / 2297127618)
 Update state (0x61) downloading, progress: 78.92 (1812803021 / 2297127618)
 Update state (0x61) downloading, progress: 89.89 (2064876438 / 2297127618)
 Update state (0x61) downloading, progress: 94.15 (2162700528 / 2297127618)
 Update state (0x61) downloading, progress: 95.68 (2197823017 / 2297127618)
 Update state (0x61) downloading, progress: 97.71 (2244495991 / 2297127618)
 Update state (0x81) verifying update, progress: 7.03 (161557428 / 2297127618)
 Update state (0x81) verifying update, progress: 32.41 (744586025 / 2297127618)
 Update state (0x81) verifying update, progress: 57.82 (1328102810 / 2297127618)
 Update state (0x81) verifying update, progress: 83.73 (1923315723 / 2297127618)
Success! App '232330' fully installed.
Server is running, waiting for SIGTERM
fetching GitHub command_start.sh...�[32mOK�[0m

�[K[ .... ] Starting gmodserver: fetching GitHub check_root.sh...�[32mOK�[0m
fetching GitHub check_system_dir.sh...�[32mOK�[0m
fetching GitHub check_executable.sh...�[32mOK�[0m
fetching GitHub check_logs.sh...�[32mOK�[0m

�[K[ .... ] Starting gmodserver: Checking for log files
�[K[�[36m INFO �[0m] Starting gmodserver: Checking for log files: Creating log files
installing log dir: /home/steam/log...�[32mOK�[0m
installing LinuxGSM log dir: /home/steam/log/script...�[32mOK�[0m
creating LinuxGSM log: /home/steam/log/script/gmodserver-script.log...�[32mOK�[0m
installing console log dir: /home/steam/log/console...�[32mOK�[0m
creating console log: /home/steam/log/console/gmodserver-console.log...�[32mOK�[0m
creating symlink to game log dir: /home/steam/log/server -> /home/steam/serverfiles/garrysmod/logs...�[32mOK�[0m
creating symlink to steam log dir: /home/steam/log/steam -> /home/steam/.steam/steam/logs...�[32mOK�[0m
fetching GitHub check_config.sh...�[32mOK�[0m
fetching GitHub check_status.sh...�[32mOK�[0m
fetching GitHub fix_steamcmd.sh...�[32mOK�[0m

�[K[ .... ] Starting gmodserver: Applying steamclient.so sdk64 hardlink fix: Garry's Mod
�[K[�[36m INFO �[0m] Starting gmodserver: Applying steamclient.so sdk64 hardlink fix: Garry's Mod
�[K[�[32m  OK  �[0m] Starting gmodserver: Applying steamclient.so sdk64 hardlink fix: Garry's Mod

�[K[ .... ] Starting gmodserver: Applying steamclient.so sdk32 link fix: Garry's Mod
�[K[�[36m INFO �[0m] Starting gmodserver: Applying steamclient.so sdk32 link fix: Garry's Mod
�[K[�[32m  OK  �[0m] Starting gmodserver: Applying steamclient.so sdk32 link fix: Garry's Mod
fetching GitHub core_logs.sh...�[32mOK�[0m

�[K[ .... ] Starting gmodserver: Example Name
�[K[�[32m  OK  �[0m] Starting gmodserver: Example Name
�[K[�[32m  OK  �[0m] Starting gmodserver: Example Name

Log: /home/steam/log/console/gmodserver-console.log

Using default binary: ./srcds_linux

Server will auto-restart if there is a crash.

mount.cfg adding path: '/home/steam/addons/css/cstrike'

mount.cfg adding path: '/home/steam/addons/css/hl2'

Couldn't load shader dll: game_shader_generic_garrysmod_srv.soConVarRef mat_dxlevel doesn't point to an existing ConVar

Game_srv.so loaded for "Garry's Mod"

Setting breakpad minidump AppID = 4000

Initializing Steam libraries for Workshop..

[S_API] SteamAPI_Init(): Loaded local 'steamclient.so' OK.

CAppInfoCacheReadFromDiskThread took 4 milliseconds to initialize

dlmopen steamservice.so failed: steamservice.so: cannot open shared object file: No such file or directory

SteamInternal_SetMinidumpSteamID:  Caching Steam ID:  76561197960265728 [API loaded no]

[S_API FAIL] Tried to access Steam interface SteamUser023 before SteamAPI_Init succeeded.

-authkey is no longer required by Garry's Mod. You can safely remove it from your server's launch options.

WS: No +host_workshop_collection or it is invalid!

Unknown command "cl_cmdrate"

Unknown command "cl_updaterate"

Unknown command "rate"

"host_workshop_collection" = ""

 game

Changing gamemode to Trouble in Terrorist Town (terrortown)

Usage: sv_setsteamaccount <login_token>

Network: IP 0.0.0.0, mode MP, dedicated Yes, ports 27016 SV / 27005 CL

PREP OK

Included TTT language file: brazilian_portuguese.lua

Included TTT language file: chef.lua

Included TTT language file: english.lua

Included TTT language file: french.lua

Included TTT language file: german.lua

Included TTT language file: italian.lua

Included TTT language file: japanese.lua

Included TTT language file: russian.lua

Included TTT language file: simpchinese.lua

Included TTT language file: spanish.lua

Included TTT language file: swedish.lua

Included TTT language file: tradchinese.lua

Included TTT language file: ukrainian.lua

Trouble In Terrorist Town gamemode initializing...

ConVarRef room_type doesn't point to an existing ConVar

Executing dedicated server config file gmodserver.cfg

Using map cycle file cfg/mapcycle.txt.

Initializing Steam libraries for secure Internet server

[S_API] SteamAPI_Init(): Loaded local 'steamclient.so' OK.

CAppInfoCacheReadFromDiskThread took 3 milliseconds to initialize

dlmopen steamservice.so failed: steamservice.so: cannot open shared object file: No such file or directory

SteamInternal_SetMinidumpSteamID:  Caching Steam ID:  76561197960265728 [API loaded no]

No account token specified; logging into anonymous game server account.  (Use sv_setsteamaccount to login to a persistent account.)

Server ranking in server browser will be negatively impacted due to sv_setsteamaccount not being set.

Server logging enabled.

Server logging data to file logs/L1130000.log

L 11/30/2023 - 16:08:27: Log file started (file "logs/L1130000.log") (game "/home/steam/serverfiles/garrysmod") (version "9040")

L 11/30/2023 - 16:08:27: server_cvar: "sv_airaccelerate" "100"

Unknown command "sv_allow_wait_command"

Unknown command "sv_max_usercmd_future_ticks"

L 11/30/2023 - 16:08:27: server_cvar: "sv_alltalk" "1"

L 11/30/2023 - 16:08:27: server_cvar: "sv_contact" "youremail@changeme.com"

Unknown command "sv_voicecodec"

L 11/30/2023 - 16:08:27: server_cvar: "sbox_noclip" "0"

L 11/30/2023 - 16:08:27: server_cvar: "sbox_weapons" "0"

L 11/30/2023 - 16:08:27: server_cvar: "sbox_playershurtplayers" "0"

L 11/30/2023 - 16:08:27: server_cvar: "sbox_maxprops" "100"

L 11/30/2023 - 16:08:27: server_cvar: "sbox_maxragdolls" "50"

L 11/30/2023 - 16:08:27: server_cvar: "sbox_maxballoons" "10"

L 11/30/2023 - 16:08:27: server_cvar: "sbox_maxeffects" "0"

L 11/30/2023 - 16:08:27: server_cvar: "sbox_maxdynamite" "0"

L 11/30/2023 - 16:08:27: server_cvar: "sbox_maxlamps" "5"

L 11/30/2023 - 16:08:27: server_cvar: "sbox_maxthrusters" "20"

L 11/30/2023 - 16:08:27: server_cvar: "sbox_maxwheels" "20"

L 11/30/2023 - 16:08:27: server_cvar: "sbox_maxhoverballs" "20"

L 11/30/2023 - 16:08:27: server_cvar: "sbox_maxvehicles" "1"

L 11/30/2023 - 16:08:27: server_cvar: "sbox_maxbuttons" "20"

L 11/30/2023 - 16:08:27: server_cvar: "sbox_maxemitters" "0"

'banned_user.cfg' not present; not executing.

'banned_ip.cfg' not present; not executing.

Unknown command "heartbeat"

L 11/30/2023 - 16:08:27: server_cvar: "sv_password" "***PROTECTED***"

L 11/30/2023 - 16:08:27: server_cvar: "sv_voiceenable" "0"

Connection to Steam servers successful.

   Public IP is xxx.xxx.xxx.xxx.

Assigned anonymous gameserver Steam ID [A-1:3105794051(24760)].

VAC secure mode is activated.


@alexstaeding
Copy link
Author

In my configuration I have all ports set to the same number: 27015. I also tried 27016 as in your example but that still doesn't explain the segfault.

Also, your container log keeps going further than mine. Mine stops here:

[ INFO ] Starting gmodserver: Checking for log files: Creating log files
installing log dir: /home/steam/log...OK
installing LinuxGSM log dir: /home/steam/log/script...OK
creating LinuxGSM log: /home/steam/log/script/gmodserver-script.log...OK
installing console log dir: /home/steam/log/console...OK
creating console log: /home/steam/log/console/gmodserver-console.log...OK
creating symlink to game log dir: /home/steam/log/server -> /home/steam/serverfiles/garrysmod/logs...OK
creating symlink to steam log dir: /home/steam/log/steam -> /home/steam/.steam/steam/logs...OK
fetching GitHub check_config.sh...OK
fetching GitHub check_status.sh...OK
fetching GitHub fix_steamcmd.sh...OK
[  OK  ] Starting gmodserver: Applying steamclient.so sdk64 hardlink fix: Garry's Mod
[  OK  ] Starting gmodserver: Applying steamclient.so sdk32 link fix: Garry's Mod
fetching GitHub core_logs.sh...OK
[  OK  ] Starting gmodserver: My Server

@jusito
Copy link
Owner

jusito commented Dec 1, 2023

Sry I missed the note that I merged docker logs with console log: /home/steam/log/console/gmodserver-console.log
You can also also access the console (on docker, don't know the k8s way) docker exec -it container console.
Please try the exact configuration I posted or give me an exact configuration I can try out, else we could miss something, even if its just "My Server" vs "Example Name" or an additional env var.

If the log until server start is identical, you have no addition warning or error, probably the issue is somewhere deeper.

  • In your log above there seems to be an issue with serverevents.res, can you check the md5sum of this file? docker exec -it debug-debug-1 md5sum /home/steam/serverfiles/garrysmod/resource/serverevents.res it should be 41149c59633dee2c9c145827e2cc3d7e /home/steam/serverfiles/garrysmod/resource/serverevents.res
  • Gmod offers a debug flag:
    • Modify the Dockerfile in Line 42 to also install the package "gdb"
    • SERVER_ADDITIONAL_PARAMETERS: "-disableluarefresh -debug -debuglog /home/steam/serverfiles/debug.log"

@alexstaeding
Copy link
Author

The md5sum of the file matches...

alex@c01:~/ttt$ kubectl exec -it -n ttt server-0 -- /bin/bash
Defaulted container "ttt-server" out of: ttt-server, fix-permissions (init)
steam@server-0:/$ md5sum /home/steam/serverfiles/garrysmod/resource/serverevents.res
41149c59633dee2c9c145827e2cc3d7e  /home/steam/serverfiles/garrysmod/resource/serverevents.res

I have tried now with a very simple docker compose configuration to debug this.

version: '3.8'

services:
  server:
    image: jusito/docker-ttt:gmod_ttt_debian
    container_name: ttt-server
    ports:
      - "27015:27015/tcp"
      - "27015:27015/udp"
    environment:
      - SERVER_PORT=27015
      - INSTALL_CSS=true
      - WORKSHOP_COLLECTION_ID=2176094350
      - SERVER_NAME=My Server
      - SERVER_PASSWORD=server-password
      - SERVER_DEFAULT_MAP=ttt_waterworld
  • It crashes on my macbook pro m2
  • It works in a debian 12 VM

I think that is a different error on my macbook, so I will open another issue.
As for this issue, I will continue trying to narrow it down - at least I have a working configuration now.

@alexstaeding
Copy link
Author

alexstaeding commented Dec 1, 2023

I discovered (at least part of) the problem. This diff fixes it:

diff --git a/kubernetes/ttt/deploy.yaml b/kubernetes/ttt/deploy.yaml
index 898d92b..5a4ec75 100644
--- a/kubernetes/ttt/deploy.yaml
+++ b/kubernetes/ttt/deploy.yaml
@@ -54,7 +54,8 @@ spec:
               value: "ttt_waterworld"
           volumeMounts:
             - name: data-volume
-              mountPath: /home/steam/serverfiles
+              mountPath: /home/steam/serverfiles/steam_cache
+              subPath: steam_cache
   volumeClaimTemplates:
     - metadata:
         name: data-volume

It seems there are some files in the /home/steam/serverfiles directory which do not enjoy being either in a kubernetes volume mount or a ceph filesystem.

@alexstaeding
Copy link
Author

alexstaeding commented Dec 1, 2023

But even mounting the steam_cache doesn't work properly. It's all being redownloaded.

Processing addon 47: FY-71 Custom SWEP for TTT (329300929)...
   Failed to open .gma, redownloading...
   Downloaded!
   Extracting...
   Mounting...
   Mounted!
Processing addon 46: ttt_terraria (337311940)...
   Failed to open .gma, redownloading...
   Downloaded!
   Extracting...
   Mounting...
   Mounted!
Processing addon 45: TTT_Bank_b3 & b13 (338284205)...
   Failed to open .gma, redownloading...
   Downloaded!
   Extracting...
   Warning! Addon was created with a 3rd party tool, which might cause install/load issues.
   Mounting...
   Mounted!
Processing addon 44: TTT Minecraft-Island (353329143)...
   Failed to open .gma, redownloading...
   Downloaded!
   Extracting...
   Warning! Addon was created with a 3rd party tool, which might cause install/load issues.
   Mounting...
   Mounted!
Processing addon 43: [TTT] ttt_pacman_v1 (379662785)...
   Failed to open .gma, redownloading...
   Downloaded!
   Extracting...
   Mounting...
   Mounted!

sigh

@alexstaeding
Copy link
Author

One more thing

Sry I missed the note that I merged docker logs with console log: /home/steam/log/console/gmodserver-console.log
You can also also access the console (on docker, don't know the k8s way) docker exec -it container console.

Is it possible to pipe the console log directly to the container log? It's kind of counter-intuitive to have to enter the container to check the logs - this is normally available directly from the container logs (e.g. kubectl logs <pod>)

@jusito jusito changed the title Container terminates with message "To start server..." segmentation fault in Kubernetes Dec 3, 2023
@jusito
Copy link
Owner

jusito commented Dec 4, 2023

It is and actually in my last project: https://github.com/jusito/LinuxGSM-Docker
So yes its possible, probably should migrate my current work to my new container, just not a priority for me.

@jusito
Copy link
Owner

jusito commented Dec 14, 2023

Still trying to figure out what the cause could be, but I am making no progress. I will keep this issue open and added a label for the workaround; if others have the same issue, the workaround could also work for them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants