Fishtest Server Setup

Server setup

1. Use a clean install of Ubuntu 18.04 (bionic), 20.04 (focal) or 22.04 (jammy), on a virtual machine, cloud instance etc.
   1. If you use VirtualBox on Windows, you will need to disable the Windows Hypervisor.
      See: https://forums.virtualbox.org/viewtopic.php?f=25&t=99390 and unplug your PC afterwards!
      More from Microsoft about this, here.
2. Copy the script setup_fishtest.sh:
   1. Write your password in the variable usr_pwd
   2. (Optional to use https) Write your fully qualified domain name in the variable server_name
3. Run the setup script using bash as:

   sudo bash setup_fishtest.sh 2>&1 | tee setup_fishtest.sh.log

Nets workflow setup

1. Download the nets required by the tests (e.g. the default one) from the official fishtest server NN Stats page
2. Open a web browser using the ip_address of the fishtest server (http://ip_address/login)
3. Login as user01 (with password user01)
4. Select the "NN Upload page" (http://ip_address/upload)
5. Upload the net. The net is written in /home/fishtest/net-server/nn
6. (Optional) Use the script /home/fishtest/net-server/set_net_server.sh to set the server (the official server or the local development server) from which to download the nets during the tests

Create tests

Use these users

• To approve test:
  • User: user00
  • Password: user00
• To create test:
  • User: user01
  • Password: user01

Connect a worker

Use the ip_address of the fishtest server
To have multiple workers make some copies of the worker folder.

python3 worker.py <username> <password> --protocol <http/https> --host <ip_address> --port <80/443/custom> --concurrency <n_cores> 

Start/stop fishtest services

• Start the services

  sudo systemctl start fishtest@{6543..6544}.service

• Stop the services

  sudo systemctl stop fishtest@{6543..6544}.service

• Restart the services

  sudo systemctl restart fishtest@{6543..6544}.service

Debug the server

Using the Pyramid Debug Toolbar

1. Login on Ubuntu
2. Use the following commands to start/stop the fishtest_dbg.service
   • Start the debug session

     sudo systemctl start fishtest_dbg.service

   • Stop the debug session

     sudo systemctl stop fishtest_dbg.service

3. Open a browser using the port 6542 (http://ip_address:6542).

---

Scripts

Server setup

For Ubuntu 18.04 (bionic), 20.04 (focal) or 22.04 (jammy)

Click to view

#!/bin/bash
# 250206
# to setup a fishtest server on Ubuntu 18.04 (bionic), 20.04 (focal) or 22.04 (jammy), simply run:
# sudo bash setup_fishtest.sh 2>&1 | tee setup_fishtest.sh.log
#
# to use fishtest connect a browser to:
# http://<ip_address> or http://<fully_qualified_domain_name>

set -euo pipefail
IFS=$'\n\t'

user_name='fishtest'
user_pwd='<your_password>'
# try to find the ip address
server_name=$(hostname --all-ip-addresses)
server_name=$(echo $server_name)
# use a fully qualified domain names (http/https)
# server_name='<fully_qualified_domain_name>'

git_user_name='your_name'
git_user_email='you@example.com'

# create user for fishtest
useradd -m -s /bin/bash ${user_name}
echo ${user_name}:${user_pwd} | chpasswd
usermod -aG sudo ${user_name}
sudo -i -u ${user_name} << EOF
mkdir .ssh
chmod 700 .ssh
touch .ssh/authorized_keys
chmod 600 .ssh/authorized_keys
EOF

# get the user $HOME
user_home=$(sudo -i -u ${user_name} << 'EOF'
echo ${HOME}
EOF
)

# add some bash variables
sudo -i -u ${user_name} << 'EOF'
cat << 'EOF0' >> .profile

export FISHTEST_HOST=127.0.0.1
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export VENV="$HOME/fishtest/server/.venv"
EOF0
EOF

# set secrets
sudo -i -u ${user_name} << EOF
echo '' > fishtest.secret
echo '' > fishtest.captcha.secret

cat << EOF0 > .netrc
# GitHub authentication to raise API rate limit
# create a <personal-access-token> https://github.com/settings/tokens
#machine api.github.com
#login <personal-access-token>
#password x-oauth-basic
EOF0
chmod 600 .netrc
EOF

# install required packages
apt update && apt full-upgrade -y && apt autoremove -y && apt clean
apt install -y bash-completion curl exim4 git gnupg mutt procps ufw

# configure ufw
ufw allow ssh
ufw allow http
ufw allow https
ufw allow 6542
ufw --force enable
ufw status verbose

# setup pyenv and install the latest python version
# https://github.com/pyenv/pyenv
apt update
apt install -y build-essential pkg-config libb2-dev libbz2-dev libffi-dev libgdbm-dev libgdbm-compat-dev liblzma-dev libncurses5-dev libreadline6-dev libsqlite3-dev libssl-dev lzma-dev tk-dev uuid-dev zlib1g-dev

sudo -i -u ${user_name} << 'EOF'
git clone https://github.com/pyenv/pyenv.git "${HOME}/.pyenv"
cat << 'EOF0' >> "${HOME}/.profile"

# pyenv: keep at the end of the file
export PYENV_ROOT="$HOME/.pyenv"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
EOF0

cat << 'EOF0' >> "${HOME}/.bashrc"

# pyenv: keep at the end of the file
export PYENV_ROOT="$HOME/.pyenv"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
EOF0
EOF

# optimized python build: LTO takes some time to run the tests and make the second optimized build
# consider to install a newer GCC or CLANG
# CONFIGURE_OPTS="--enable-optimizations --with-lto" MAKE_OPTS="--jobs 2" PYTHON_CFLAGS="-march=native -mtune=native" pyenv install ${python_ver}
python_ver="3.13.2"
sudo -i -u ${user_name} << EOF
pyenv install ${python_ver}
pyenv global ${python_ver}
EOF

# install mongodb community edition for Ubuntu 20.04 and 22.04 (jammy)
curl -fsSL https://www.mongodb.org/static/pgp/server-7.0.asc | sudo gpg -o /usr/share/keyrings/mongodb-server-7.0.gpg --dearmor
ubuntu_release=$(lsb_release -c | awk '{print $2}')
echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-7.0.gpg ] https://repo.mongodb.org/apt/ubuntu ${ubuntu_release}/mongodb-org/7.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-7.0.list
apt update
apt install -y mongodb-org

# set the cache size in /etc/mongod.conf
#  wiredTiger:
#    engineConfig:
#      cacheSizeGB: 2.00
cp /etc/mongod.conf mongod.conf.bkp
sed -i 's/^#  wiredTiger:/  wiredTiger:\n    engineConfig:\n      cacheSizeGB: 2.00/' /etc/mongod.conf
# set the memory decommit
sed -i '/^## Enterprise-Only Options:/i\setParameter:\n  tcmallocAggressiveMemoryDecommit: 1\n' /etc/mongod.conf
# setup logrotate for mongodb
sed -i '/^  logAppend: true/a\  logRotate: reopen' /etc/mongod.conf

cat << 'EOF' > /etc/logrotate.d/mongod
/var/log/mongodb/mongod.log
{
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0600 mongodb mongodb
    sharedscripts
    postrotate
        /bin/kill -SIGUSR1 $(pgrep mongod 2>/dev/null) 2>/dev/null || true
    endscript
}
EOF

# download fishtest
sudo -i -u ${user_name} << EOF
git clone --single-branch --branch master https://github.com/official-stockfish/fishtest.git
cd fishtest
git config user.email "${git_user_email}"
git config user.name "${git_user_name}"
EOF

# setup fishtest
sudo -i -u ${user_name} << 'EOF'
python3 -m venv ${VENV}
${VENV}/bin/python3 -m pip install --upgrade pip setuptools wheel
cd ${HOME}/fishtest/server
${VENV}/bin/python3 -m pip install --use-pep517 -e .
EOF

# install fishtest as systemd service
cat << EOF > /etc/systemd/system/fishtest@.service
[Unit]
Description=Fishtest Server port %i
After=network.target mongod.service

[Service]
Type=simple
ExecStart=${user_home}/fishtest/server/.venv/bin/pserve production.ini http_port=%i
Restart=on-failure
RestartSec=3
User=${user_name}
WorkingDirectory=${user_home}/fishtest/server

[Install]
WantedBy=multi-user.target
EOF

# install also fishtest debug as systemd service
cat << EOF > /etc/systemd/system/fishtest_dbg.service
[Unit]
Description=Fishtest Server Debug port 6542
After=network.target mongod.service

[Service]
Type=simple
ExecStart=${user_home}/fishtest/server/.venv/bin/pserve development.ini --reload
User=${user_name}
WorkingDirectory=${user_home}/fishtest/server

[Install]
WantedBy=multi-user.target
EOF

# enable the autostart for mongod.service and fishtest@.service
# check the log with: sudo journalctl -u fishtest@6543.service --since "2 days ago"
systemctl daemon-reload
systemctl enable mongod.service
systemctl enable fishtest@{6543..6545}.service

# start fishtest server
systemctl start mongod.service
systemctl start fishtest@{6543..6545}.service

# add mongodb indexes
sudo -i -u ${user_name} << 'EOF'
${VENV}/bin/python3 ${HOME}/fishtest/server/utils/create_indexes.py actions flag_cache pgns runs users
EOF

# add some default users:
# "user00" (with password "user00"), as approver
# "user01" (with password "user01"), as normal user
sudo -i -u ${user_name} << 'EOF'
${VENV}/bin/python3 << EOF0
from fishtest.rundb import RunDb
rdb = RunDb()
for i in range(10):
    user_name = f"user{i:02d}"
    user_mail = f"{user_name}@example.org"
    user_repo = "https://github.com/official-stockfish/Stockfish"
    rdb.userdb.create_user(user_name, user_name, user_mail, user_repo)
    if i == 0:
        rdb.userdb.add_user_group(user_name, "group:approvers")
    user = rdb.userdb.get_user(user_name)
    user["blocked"] = False
    user["pending"] = False
    user["machine_limit"] = 100
    rdb.userdb.save_user(user)
EOF0
EOF

sudo -i -u ${user_name} << 'EOF'
(crontab -l; cat << EOF0
VENV=${HOME}/fishtest/server/.venv
UPATH=${HOME}/fishtest/server/utils

# Backup mongodb database and upload to s3
# keep disabled on dev server
# 3 */6 * * * /usr/bin/nice -n 10 -- sh \${UPATH}/backup.sh

# Update the users table
# 1,16,31,46 * * * * /usr/bin/nice -n 10 -- \${VENV}/bin/python3 \${UPATH}/delta_update_users.py

# Purge old pgn files
# 33 3 * * *  /usr/bin/nice -n 10 -- \${VENV}/bin/python3 \${UPATH}/purge_pgns.py

# Clean up old mail (more than 9 days old)
# 33 5 * * * /usr/bin/nice -n 10 screen -D -m mutt -e 'push D~d>9d<enter>qy<enter>'

# Backup new nets on aws s3
# keep disabled on dev server
# 5 */6 * * * /usr/bin/nice -n 10 -- \${NPATH}/.venv/bin/python3 \${UPATH}/aws_nets_sync.py --backup

# Verify nets hashes
# 35 2 * * * /usr/bin/nice -n 10 -- \${VENV}/bin/python3 \${UPATH}/aws_nets_sync.py --check

EOF0
) | crontab -
EOF

# create folder tree for nginx
mkdir -p /var/www/fishtest/nn
chown ${user_name}:${user_name} /var/www/fishtest/nn
ln -sf ${user_home}/fishtest/server/fishtest/static  /var/www/fishtest/static

# install mainline nginx packages
# http://nginx.org/en/linux_packages.html#Ubuntu
apt install -y curl gnupg2 ca-certificates lsb-release ubuntu-keyring
curl https://nginx.org/keys/nginx_signing.key | gpg --dearmor | tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null
echo "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg arch=amd64] http://nginx.org/packages/mainline/ubuntu `lsb_release -cs` nginx" | sudo tee /etc/apt/sources.list.d/nginx.list
echo -e "Package: *\nPin: origin nginx.org\nPin: release o=nginx\nPin-Priority: 900\n" | sudo tee /etc/apt/preferences.d/99nginx
apt update && apt install -y nginx
mkdir -p /etc/nginx/{sites-available,sites-enabled}

# configure nginx
# check connections: netstat -anp | grep python3 | grep ESTAB | wc -l
cp /etc/nginx/nginx.conf nginx.conf.bkp
# use ubuntu default user for nginx
sed -i 's/^user  nginx;/user  www-data;/' /etc/nginx/nginx.conf
# change worker_processes from auto to 2
sed -i 's/^worker_processes  auto;/worker_processes  2;/' /etc/nginx/nginx.conf
# add worker_rlimit_nofile 8192;
#sed -i '/^pid        \/var\/run\/nginx.pid;/a\\nworker_rlimit_nofile 8192;' /etc/nginx/nginx.conf
# raise worker_connections from 1024 to 4096
#sed -i 's/worker_connections  1024;/worker_connections  4096;/' /etc/nginx/nginx.conf
# enable gzip
sed -i 's/^    #gzip  on;/    gzip  on;/' /etc/nginx/nginx.conf
# load site-enabled
sed -i '/^    include \/etc\/nginx\/conf.d\/\*.conf;/a\    include \/etc\/nginx\/sites-enabled\/*.conf;' /etc/nginx/nginx.conf
# log upstream response time
sed -i 's/\("\$http_user_agent" "\$http_x_forwarded_for"\)/\1 $upstream_response_time/' /etc/nginx/nginx.conf

cat << EOF > /etc/nginx/sites-available/fishtest.conf
upstream backend_6543 {
    server 127.0.0.1:6543;
    keepalive 64;
}

upstream backend_6544 {
    server 127.0.0.1:6544;
    keepalive 64;
}

upstream backend_6545 {
    server 127.0.0.1:6545;
    keepalive 64;
}

map \$uri \$backends {
    /tests                                    backend_6544;
    ~^/api/(actions|active_runs|calc_elo)     backend_6545;
    ~^/api/(nn/|pgn/|run_pgns/|upload_pgn)    backend_6545;
    ~^/tests/(finished|machines|user)         backend_6545;
    ~^/(actions/|contributors)                backend_6545;
    ~^/(api|tests)/                           backend_6543;
    default                                   backend_6544;
}

server {
    listen 80;
    listen [::]:80;

    server_name ${server_name};

    location ~ ^/(css/|html/|img/|js/|favicon.ico\$|robots.txt\$) {
        root        /var/www/fishtest/static;
        expires     1y;
        add_header  Cache-Control public;
        access_log  off;
    }

    location /nn/ {
        root         /var/www/fishtest;
        gzip_static  always;
        gunzip       on;
    }

    location / {
        rewrite ^/$       /tests permanent;
        rewrite ^/tests/$ /tests permanent;
        proxy_set_header Connection "";
        proxy_set_header X-Forwarded-Proto  \$scheme;
        proxy_set_header X-Forwarded-For    \$proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host   \$host:\$server_port;
        proxy_set_header X-Forwarded-Port   \$server_port;

        client_max_body_size        120m;
        client_body_buffer_size     128k;
        proxy_connect_timeout       5s;
        proxy_send_timeout          30s;
        proxy_read_timeout          30s;
        proxy_max_temp_file_size    0;
        proxy_redirect              off;
        proxy_http_version          1.1;

        gunzip on;

        proxy_pass http://\$backends;
    }
}
EOF

mv /etc/nginx/conf.d/*.conf /etc/nginx/sites-available
ln -sf /etc/nginx/sites-available/fishtest.conf /etc/nginx/sites-enabled/fishtest.conf

# create cidr.conf
${user_home}/fishtest/server/.venv/bin/python3 ${user_home}/fishtest/server/utils/nginx_cidr_builder.py --output /etc/nginx/conf.d/cidr.conf

# set number of file limit for a service managed by systemd
# interactive way: sudo systemctl edit nginx
# check limits with:
# pgrep nginx
# cat /proc/<PID>/limits
#mkdir -p /etc/systemd/system/nginx.service.d
#cat << EOF > /etc/systemd/system/nginx.service.d/override.conf
#[Service]
#LimitNOFILE=16384
#EOF

# restart nginx
usermod -aG ${user_name} www-data
systemctl daemon-reload
systemctl enable nginx.service
systemctl restart nginx.service

cat << EOF
connect a browser to:
http://${server_name}
EOF

Let's Encrypt setup

For Ubuntu 18.04 (bionic), 20.04 (focal) or 22.04 (jammy)

Click to view

# sudo bash setup-certbot.sh 2>&1 | tee setup-certbot.sh.log
# install certbot to setup let's encrypt
# https://certbot.eff.org/
# requires a DNS and a fully qualified domain name as servername
snap install core
snap refresh core
snap install --classic certbot
ln -s /snap/bin/certbot /usr/bin/certbot
cat << EOF
to configure let's encrypt run:
sudo certbot --nginx
EOF

Test a PR/branch

Server

#!/bin/bash
# to update a fishtest server simply run:
# sudo bash update_fishtest.sh 2>&1 | tee update_fishtest.sh.log
#
# to use fishtest connect a browser to:
# http://<ip_address>
user_name='fishtest'

echo "previous requirements"
sudo -i -u ${user_name} << 'EOF'
${VENV}/bin/python3 -m pip list
EOF

systemctl stop cron
systemctl stop fishtest@{6543..6545}

# download and prepare fishtest
sudo -i -u ${user_name} << EOF
rm -rf fishtest
git clone --single-branch --branch master https://github.com/official-stockfish/fishtest.git
cd fishtest
git config user.email 'you@example.com'
git config user.name 'your_name'
# add here the upstream branch to be tested
#git remote add <your_upstream> https://github.com/<your_username>/fishtest
#git pull --no-edit --rebase <your_upstream> <your_branch>
# add here the PR/PRs to be tested
#git pull --no-edit --rebase origin pull/<PR_number>/head
#git pull --no-edit --rebase origin pull/<PR_number>/head
# setup fishtest
sudo -i -u ${user_name} << 'EOF'
python3 -m venv ${VENV}
${VENV}/bin/python3 -m pip install --upgrade pip setuptools wheel
cd ${HOME}/fishtest/server
${VENV}/bin/python3 -m pip install --use-pep517 -e .
EOF
# start fishtest
systemctl start cron
systemctl start fishtest@{6543..6545}
cat << EOF
connect a browser to:
http://$(hostname --all-ip-addresses)/tests
EOF

Worker

#!/bin/bash
# requirements:
# sudo apt update && sudo apt install -y python3 python3-pip python3-venv git build-essential libnuma-dev

test_folder=/<full_path>/__test_folder
virtual_env=${test_folder}/.venv

rm -rf ${test_folder}
mkdir -p ${test_folder}
cd ${test_folder}
git clone --single-branch --branch master https://github.com/official-stockfish/fishtest.git
cd fishtest
git config user.email "you@example.com"
git config user.name "your_name"

# add here the upstream branches to be tested
#git remote add <upstream_0> https://github.com/<username_0>/fishtest
#git pull --no-edit --rebase <upstream_0> <branch_0>
#git remote add <upstream_1> https://github.com/<username_1>/fishtest
#git pull --no-edit --rebase <upstream_1> <branch_1>

# add here the PRs to be tested
#git pull --no-edit --rebase origin pull/<PR_number_0>/head
#git pull --no-edit --rebase origin pull/<PR_number_1>/head

cd worker
python3 -m venv ${virtual_env}
${virtual_env}/bin/python3 -m pip install --upgrade pip setuptools wheel

${virtual_env}/bin/python3 worker.py user00 user00 --protocol http --host <ip-address> --port 80 --concurrency 3 

Mongodb: backup and restore

Click to view

Use the mongodb tools in a temporary folder, e.g.

• Backup

  mongodump --gzip && tar -cvf dump.tar dump && rm -rf dump
  

• Restore

  tar -xvf dump.tar && mongorestore --gzip --drop && rm -rf dump
  

Stop fishtest and cron services before a mongodb restore:

sudo systemctl stop fishtest@{6543..6545}
sudo systemctl stop cron

---

FAQ

Bad worker

Sometime a badly configured worker client may post lots of losses on time during tests, or cause some tests to stop
The best policy to follow in these cases would be:

1. Login with your own approver or user username/password
2. Click on a worker name in either the Events Log or the Workers table
3. All the workers names are now a hyperlink
4. You can block/unblock the worker. If you are an approver continue with the other steps
5. Write in the Message box the reason for the block and some short instruction to solve the issue. If blocked, a worker is not able to get a new task and it shows the Message
6. Raise your concerns about the worker anomalous behavior in the appropriate channel inside the Discord server
7. If necessary, write an email to the user asking to control the worker

Block a user

Approvers (and only approvers) can now block malicious users on fishtest:

1. Login with your own approver username/password
2. Click on a run to see the test page
3. All the workers names are now a hyperlink for the approvers
4. Click on a worker name (field "username")
5. You view some info about the user (e.g. the email)
6. You can block/unblock the user. If blocked, the user cannot login in fishtest (e.g. submitting a test) and the workers cannot join the framework