Add configuration for Registry

[jpmckinney]

• All the files are in the /data/deploy on amy01.
• Links to secret gists in this issue: https://github.com/open-contracting/deploy-pillar-private/issues/9

[jpmckinney]

grepping .bash_history and .zsh_history in /home/datlab, and ~root/.bash_history, manually installed packages include the following.

Alternatively, we can find manually installed packages with: https://ocdsdeploy.readthedocs.io/en/latest/develop/maintain.html#list-manually-installed-packages

comm -23 <(apt-mark showmanual | sort -u) <(curl -sS https://releases.ubuntu.com/bionic/ubuntu-18.04.5-live-server-amd64.manifest | cut -f1 | cut -d: -f1 | sort -u)

I'm leaving these as they are potentially needed, and not problematic.

• pip
• python
• python-pip
• python2
• python3
• python3-pip
• python3-venv
• zip

---

Would be/Were installed by Salt:

• apache2 (apache)
• postgresql-12 (postgres)
• postgresql-common (postgres)
• postgresql-client-12 (postgres)
• postgresql-client-common (postgres)
• containerd.io (docker)
• docker-ce (docker)
• docker-ce-cli (docker)
• smartmontools (prometheus/node_exporter) Prometheus: Add registry server #276
• git (core/init)
• htop (core/customization)
• iotop (core/customization)
• tmux (core/customization)
• unzip (core/customization)
• vim (core/customization)
• lm-sensors (maintenance/hardware_sensors)
• wget (maintenance/rkhunter)

The following were already installed with the OS:

• bash-completion

I've removed but not purged:

• postgresql (associated with postgresql-13, which had been removed before I started)

I've purged these as unused (were used during different deployment attempts by Datlab):

• nodejs (not used)
• awscli (not used)

I've purged these as we have alternatives:

• docker-compose (1.29.2 is installed from source)
• certbot (use md module)
• python3-certbot-apache (use md module)
• ack (use grep)
• byobu (use tmux)
• mc (use vim)
• ufw (use iptables)
• screen (use tmux)
• zsh (use bash)

Other things that were (attempted to be) installed:

• ack-grep (virtual package)
• scrapyd (non-existent)

[jpmckinney]

Some notes from call with Datlab:

• Need to control log size e.g. via Docker configuration. While we could write logs to the main disk and use logrotate, doing so will make it hard to read the logs, as multiple workers will write to the same file.
  • Switched to local file logging driver, which performs rotation
• When ready, schedule cbom to run every 5 mins, so that it schedules new tasks regularly (main concern to having it run always is when there are a lot of publications without data - it will create a high load)

Run Kingfisher Collect outsider Docker (simpler access to logs, easier to re-configure, easier spider deployment)

• Configure Salt
• Clean up /data/deploy
• Remove the container
• Remove the image

Remove the pelican-frontend Vue service (the pelican-frontend Django service is needed as it's the API to the backend)

• Update docker-compose.yaml
• Clean up /data/deploy
• Remove the container
• Remove the image

Run Rabbit outside Docker (easier to re-configure):

• Delete configuration from pelican-backend
• Scale non-web workers to 0
• Check that queues are empty (purge if needed)
• Make note of the exchanges and queues
• Delete the Rabbit container
• Install and configure RabbitMQ in Salt
• Check that exchanges and queues are created

Notes from call with Yohanna: moved to #307 https://github.com/open-contracting/deploy/issues/306 and #308

[jpmckinney]

Moved to #200 (comment)

[jpmckinney]

Related issues: open-contracting/data-registry#123

[jpmckinney]

My other notes:

• Configure Salt for PostgreSQL
• Configure Salt for RabbitMQ
• Configure Salt for Apache
• Remove certbot and any related Apache configuration
• Create a new user to run docker compose
• Check Fathom integration (that localhost is excluded, etc.) since Fathom seems to be receiving hits
• Review docker logs for errors (grep dqt.log for non-info, non-debug, and replace the file with the output)

Cleanup:

• Remove containers for Kingfisher Collect from GitHub
• Remove the datlab user from the server, once it's running no processes find / -type d -not -path '/data/storage/kingfisher-collect/*' \( -user datlab -o -group datlab \)
  • Spoonbill data: /data/spoonbill
  • Spoonbill processes: /proc
  • Spoonbill containers: /var/lib/docker/overlay2

Other repositories:

• Review CI setups across pelican-frontend, pelican-backend, data-registry, kingfisher-collect, kingfisher-process
• Get registry, pelican, etc. to work in development without having to use --settings
• Eliminate deployment-specific code from individual repositories

[yolile]

Configure Salt for RabbitMQ

I cannot find the config for deploying RabbitMQ anywhere in this repo. And the data registry is using a user that doesn't exist: https://github.com/open-contracting/deploy-pillar-private/blob/f28e1de09ec4c322567cb36e0c5e645cd9abfb9f/registry.sls#L4 so I manually created that user with that password in the server for now. Update: config added here https://github.com/open-contracting/deploy-pillar-private/pull/17

[yolile]

We also need to:

• Run the migrations for pelican-backend somehow
• Create a superuser for the data_registry the first time

[jpmckinney]

Configure Salt for RabbitMQ

I cannot find the config for deploying RabbitMQ anywhere in this repo. And the data registry is using a user that doesn't exist: https://github.com/open-contracting/deploy-pillar-private/blob/f28e1de09ec4c322567cb36e0c5e645cd9abfb9f/registry.sls#L4 so I manually created that user with that password in the server for now. Update: config added here open-contracting/deploy-pillar-private#17

The Salt part is https://github.com/open-contracting/deploy/blob/main/salt/rabbitmq/init.sls The Pillar part I had forgotten to commit (with a better username and strong password). I'll make a new PR for the private part.

[jpmckinney]

We also need to:

• Run the migrations for pelican-backend somehow

Can you follow the same pattern as the data registry?

• Create a superuser for the data_registry the first time

Do we need this for Salt or other deployment actions? If not, we typically do this manually.

[yolile]

Can you follow the same pattern as the data registry?

In the data registry, the django migration command is run as a separate container. However, as far as I understand in pelican-backend that command is not used but the migration files are executed directly with psql the first time, or maybe I'm missing something?

Do we need this for Salt or other deployment actions? If not, we typically do this manually.

Ok

[yolile]

• Use RABBIT_URL instead of HOST and PORT when chore: Consolidate settings into one file pelican-frontend#60 is merged

[jpmckinney]

For pelican-backend, need to populate ./tools/exchange_rates_dump.csv

[jpmckinney]

For the remaining checkbox about pelican-backend's database, I'll do that in #322