Kingfisher Process Docker app deploy

docs/conf.py

@@ -61,7 +61,7 @@
     r'^https://(?:account|dcc)\.godaddy\.com',
     r'^https://(?:crm|redash)\.open-contracting\.org',
     r'^https://(?:docs\.google\.com/(?:document|spreadsheets)/d|drive\.google\.com/drive/folders)/',
-    r'^https://(?:ocp-library\.herokuapp|robot\.your-server|us-east-1\.console\.aws\.amazon)\.com',
+    r'^https://(?:ocp-library\.herokuapp|robot\.hetzner|us-east-1\.console\.aws\.amazon)\.com',
     r'^https://(?:postmaster|search)\.google\.com',
     r'^https://app\.(dmarcanalyzer|usefathom)\.com',
     r'^https://github\.com/open-contracting/[^/]+/issues/new',

docs/deploy/create_server.rst

@@ -281,19 +281,15 @@ Some IDs might fail (`#156 <https://github.com/open-contracting/deploy/issues/15
 #. If the server runs a database like PostgreSQL (``pg_dump``), MySQL (``mysqldump``) or Elasticsearch, copy the database
 #. If the server runs a web server like Apache or application server like uWSGI, optionally copy the log files
 
-Django applications
+Data support server
 ~~~~~~~~~~~~~~~~~~~
 
-#. Copy the ``media`` directory and the ``db.sqlite3`` file from the app's directory
+See :doc:`data-support`.
 
-Kingfisher
-~~~~~~~~~~
-
-.. note::
-
-   Instructions are incomplete.
+Django applications
+~~~~~~~~~~~~~~~~~~~
 
-#. Update the IP addresses in the ``pillar/tinyproxy.sls`` file, and deploy the ``docs`` service
+#. Copy the ``media`` directory and the ``db.sqlite3`` file from the app's directory
 
 OCDS documentation
 ~~~~~~~~~~~~~~~~~~

docs/deploy/data-support.rst

@@ -0,0 +1,157 @@
+Data support tasks
+==================
+
+#. Notify RBC Group of the new domain name for the new PostgreSQL server
+#. Add or update the [Redash data source](https://redash.open-contracting.org/data_sources) for the new PostgreSQL server
+
+Create a data support replica server
+------------------------------------
+
+#. Update ``postgres.replica_ipv4`` (and ``postgres.replica_ipv6``, if applicable) in the ``pillar/kingfisher_main.sls`` file.
+
+Create a data support main server
+---------------------------------
+
+#. Adjust reserved disk space to 1% for large disks:
+
+   .. code-block:: bash
+
+      tune2fs -m 1 /dev/md2
+
+Dependent services
+~~~~~~~~~~~~~~~~~~
+
+#. Update the allowed IP addresses in the ``pillar/tinyproxy.sls`` file
+#. Deploy the ``docs`` service, when ready
+
+If applicable:
+
+#. Update the allowed IP addresses and hostname in the ``pillar/kingfisher_replica.sls`` file
+#. Deploy the ``kingfisher-replica`` service, when ready
+
+Docker
+~~~~~~
+
+#. Check that ``docker.uid`` in the ``pillar/kingfisher_main.sls`` file matches the entry in the ``/etc/passwd`` file.
+
+Docker apps
+~~~~~~~~~~~
+
+#. Run migrations for :doc:`Docker apps<docker>` as the ``deployer`` user:
+
+   .. code-block:: bash
+
+      su - deployer
+
+      cd /data/deploy/kingfisher-process/
+      docker compose run --rm web python manage.py migrate
+
+      cd /data/deploy/pelican-frontend/
+      docker compose run --rm web python manage.py migrate
+
+#. :doc:`Pull new images and start new containers for each Docker app<docker>`.
+
+Pelican backend
+~~~~~~~~~~~~~~~
+
+.. warning::
+
+   A cron job updates the ``exchange_rates`` table every 12 hours. Complete this step before the next run.
+
+The initial migrations for Pelican backend are run by Salt.
+
+#. Connect to the old server, and dump the ``exchange_rates`` table:
+
+   .. code-block:: bash
+
+      sudo -i -u postgres psql -c '\copy exchange_rates (valid_on, rates, created, modified) to stdout' pelican_backend > exchange_rates.csv
+
+#. Copy the database dump to your local machine. For example:
+
+   .. code-block:: bash
+
+      rsync -avz root@ocp13.open-contracting.org:~/exchange_rates.csv .
+
+#. Copy the database dump to the new server. For example:
+
+   .. code-block:: bash
+
+      rsync -avz exchange_rates.sql root@ocp23.open-contracting.org:~/
+
+#. Populate the ``exchange_rates`` table:
+
+   .. code-block:: bash
+
+      psql -U pelican_backend -h localhost -c "\copy exchange_rates (valid_on, rates, created, modified) from 'exchange_rates.csv';" pelican_backend
+
+Kingfisher Collect
+~~~~~~~~~~~~~~~~~~
+
+#. Connect to the new server, switch to the ``incremental`` user, generate an SSH key pair, and get the public SSH key:
+
+      .. code-block:: bash
+
+         su - incremental
+         ssh-keygen -t rsa -b 4096 -C "incremental"
+         cat ~/.ssh/id_rsa.pub
+
+#. Add the public SSH key to the ``ssh.incremental`` list in the ``pillar/kingfisher_main.sls`` file:
+
+   .. code-block:: yaml
+
+      ssh:
+        incremental:
+          - ssh-rsa AAAB3N...
+
+#. Change ``cron.present`` to ``cron.absent`` in the ``salt/kingfisher/collect/incremental.sls`` file.
+#. :doc:`Deploy the old server and the new server<deploy>`.
+#. Connect to the old server:
+
+   #. Switch to the ``incremental`` user:
+
+      .. code-block:: bash
+
+         su - incremental
+
+   #. Stop any processes started by the cron jobs.
+   #. Dump the ``kingfisher_collect`` database:
+
+      .. code-block:: bash
+
+         pg_dump -U kingfisher_collect -h localhost -f kingfisher_collect.sql kingfisher_collect
+
+#. Connect to the new server:
+
+   #. Switch to the ``incremental`` user:
+
+      .. code-block:: bash
+
+         su - incremental
+
+   #. Copy the database dump from the old server. For example:
+
+      .. code-block:: bash
+
+         rsync -avz incremental@ocp04.open-contracting.org:~/kingfisher_collect.sql .
+
+   #. Load the database dump:
+
+      .. code-block:: bash
+
+         psql -U kingfisher_collect -h localhost -f kingfisher_collect.sql kingfisher_collect
+
+   #. Copy the ``data`` directory from the old server. For example:
+
+      .. code-block:: bash
+
+         rsync -avz incremental@ocp04.open-contracting.org:/home/incremental/data/ /home/incremental/data/
+
+   #. Copy the ``logs`` directory from the old server. For example:
+
+      .. code-block:: bash
+
+         rsync -avz incremental@ocp04.open-contracting.org:/home/incremental/logs/ /home/incremental/logs/
+
+#. Remove the public SSH key from the ``ssh.incremental`` list in the ``pillar/kingfisher_main.sls`` file.
+#. Change ``cron.absent`` to ``cron.present`` in the ``salt/kingfisher/collect/incremental.sls`` file.
+#. :doc:`Deploy the new server<deploy>`.

docs/deploy/deploy.rst

@@ -56,7 +56,7 @@ To `run a specific SLS ID <https://docs.saltproject.io/en/latest/ref/modules/all
 
    .. code-block:: bash
 
-      ./run.py --state-output=changes 'kingfisher-process' state.sls postgres,postgres.backup,kingfisher.collect.database,kingfisher.process.database
+      ./run.py --state-output=changes 'kingfisher-main' state.sls postgres,postgres.backup
 
 2. Check Salt output
 --------------------

docs/deploy/docker.rst

@@ -1,6 +1,10 @@
 Docker tasks
 ============
 
+.. note::
+
+   The commands assume Compose V2, which uses ``docker compose``. For Compose V1, use ``docker-compose`` (with hyphen).
+
 Change to the application's directory, replacing ``APP``:
 
 .. code-block:: bash
@@ -11,20 +15,20 @@ Change to the application's directory, replacing ``APP``:
 
 .. code-block:: bash
 
-   docker-compose pull
-   docker-compose up -d
+   docker compose pull
+   docker compose up -d
 
 Create a superuser:
 
 .. code-block:: bash
 
-   docker-compose run --rm web python manage.py createsuperuser
+   docker compose run --rm web python manage.py createsuperuser
 
 Migrate the database:
 
 .. code-block:: bash
 
-   docker-compose run --rm web python manage.py migrate
+   docker compose run --rm web python manage.py migrate
 
 .. note::
 
@@ -39,7 +43,7 @@ For example:
 
 .. code-block:: bash
 
-   psql -c 'SET ROLE pelican_backend' -c "\copy exchange_rates (valid_on, rates, created, modified) from '/opt/pelican-backend/exchange_rates.csv' delimiter ',' csv header;" pelican_backend
+   psql -U pelican_backend -h localhost -c "\copy exchange_rates (valid_on, rates, created, modified) from 'exchange_rates.csv';" pelican_backend
 
 Check that the ID sequence is correct:
 

docs/deploy/index.rst

@@ -18,7 +18,7 @@ Deployment Guides
    docker.rst
    docs.rst
    prometheus.rst
-   kingfisher.rst
+   data-support.rst
    redash.rst
    redmine.rst
 

docs/deploy/redash.rst

@@ -70,7 +70,7 @@ Load the new server's database
 
    .. code-block:: bash
 
-      sudo -u postgres psql -f redash.sql redash
+      sudo -i -u postgres psql -f redash.sql redash
 
 #. Change to the non-root user and Redash directory:
 
@@ -129,8 +129,3 @@ Upgrade the Redash service
       docker-compose up -d
 
 #. Check that the new version is running by viewing the `System Status <https://redash.open-contracting.org/admin/status>`__ page and reading the *Version*. You may need to log in to Redash to access this page.
-
-Troubleshoot
-~~~~~~~~~~~~
-
-To troubleshoot ``docker-compose`` commands, add the ``--verbose`` option.

docs/deploy/setup.rst

@@ -52,12 +52,10 @@ Ensure you have the correct versions of dependencies. Activate your virtual envi
 
    .. code-block:: bash
 
-      for i in root ocdskfp; do echo $i; su $i -c "tmux ls"; done
+      for i in root $(ls -1 /home); do echo $i; su $i -c "tmux ls"; done
 
 #. If the ``postgres`` service would be restarted by the deployment (for example, due to a configuration change or a package upgrade), check if any :ref:`long-running queries<pg-stat-activity>` are running. If there are queries with a ``state`` of ``active`` and a ``time`` greater than an hour, don't deploy without the consent of the data support managers, who should be identified by the ``usename``, ``client_addr`` or comment at the start of ``query``.
 
-If you must deploy while spiders are running, see how to :ref:`deploy Kingfisher Process without losing Scrapy requests<deploy-kingfisher-process>`.
-
 4. Get deploy token
 ~~~~~~~~~~~~~~~~~~~
 

docs/develop/get_started.rst

@@ -45,7 +45,7 @@ You must first have access to three private repositories. Contact an owner of th
 
 .. tip::
 
-   To generate new SSH keys (if they do not already exist):
+   To generate an SSH key pair (if they do not already exist):
 
    .. code-block:: bash
 
@@ -88,6 +88,6 @@ This overwrites the files:
 
    On macOS, you might need to move ``Saltfile`` to ``~/.salt/Saltfile``.
 
-This script assumes your SSH keys are ``~/.ssh/id_rsa.pub`` and ``~/.ssh/id_rsa``.
+This script assumes your SSH key pair is ``~/.ssh/id_rsa.pub`` and ``~/.ssh/id_rsa``.
 
 You're now ready to :doc:`../deploy/deploy`.