Add a dockerfile for running a set of Synapse worker processes

changelog.d/9162.misc

@@ -0,0 +1 @@
+Add a dockerfile for running Synapse in worker-mode under Complement.

docker/Dockerfile-workers

@@ -0,0 +1,23 @@
+# Inherit from the official Synapse docker image
+FROM matrixdotorg/synapse
+
+# Install deps
+RUN apt-get update
+RUN apt-get install -y supervisor redis nginx
+
+# Remove the default nginx sites
+RUN rm /etc/nginx/sites-enabled/default
+
+# Copy the worker process and log configuration files
+COPY ./docker/conf/worker.yaml.j2 /conf/worker.yaml.j2
+
+# Expose nginx listener port
+EXPOSE 8080/tcp
+
+# Volume for user-editable config files, logs etc.
+VOLUME ["/data"]
+
+# A script to read environment variables and create the necessary
+# files to run the desired worker configuration. Will start supervisord.
+COPY ./docker/configure_workers_and_start.py /configure_workers_and_start.py
+ENTRYPOINT ["/configure_workers_and_start.py"]

docker/README.md

@@ -53,6 +53,7 @@ The following environment variables are supported in `generate` mode:
 * `SYNAPSE_SERVER_NAME` (mandatory): the server public hostname.
 * `SYNAPSE_REPORT_STATS` (mandatory, `yes` or `no`): whether to enable
   anonymous statistics reporting.
+* `SYNAPSE_HTTP_PORT`: the port Synapse should start listen for http traffic on.
 * `SYNAPSE_CONFIG_DIR`: where additional config files (such as the log config
   and event signing key) will be stored. Defaults to `/data`.
 * `SYNAPSE_CONFIG_PATH`: path to the file to be generated. Defaults to
@@ -73,6 +74,8 @@ docker run -d --name synapse \
     matrixdotorg/synapse:latest
 ```
 
+assuming 8008 is the port Synapse is listening for traffic on.
+
 You can then check that it has started correctly with:
 
 ```
@@ -208,4 +211,57 @@ healthcheck:
 ## Using jemalloc
 
 Jemalloc is embedded in the image and will be used instead of the default allocator.
-You can read about jemalloc by reading the Synapse [README](../README.md)
+You can read about jemalloc by reading the Synapse [README](../README.md)
+
+### Running all worker processes in a single container
+
+Anyone looking to run Synapse workers with docker in production should use one container
+per Synapse process, specifying a different worker config per container.
+Running all Synapse processes is possible, but is currently only used for testing 
+purposes. [Dockerfile-workers](Dockerfile-workers) will produce an image bundling
+all necessary components together for a workerised homeserver instance.
+
+This includes any desired Synapse worker processes, a nginx to route traffic accordingly,
+a redis for worker communication and a supervisord instance to start up and monitor all
+processes. You will need to provide your own postgres container to connect to. TLS is not
+handled by the container. A external reverse proxy should be placed in front the exposed
+HTTP port for the purposes of TLS termination.
+
+Note that the worker Dockerfile is based off the main `[Dockerfile](Dockerfile)`. If you
+would like to build from the current checkout, first build the main Synapse docker image
+using the instructions in [Building the image](#building-the-image), then afterwards build
+the worker image:
+
+```
+docker build -t matrixdotorg/synapse:workers -f docker/Dockerfile-workers .
+```
+
+To start a container, use the following:
+
+```
+docker run -d --name synapse \
+    --mount type=volume,src=synapse-data,dst=/data \
+    -p 8008:8008 \
+    -e SYNAPSE_SERVER_NAME=my.matrix.host \
+    -e SYNAPSE_REPORT_STATS=yes \
+    -e SYNAPSE_WORKERS=synchrotron,media_repository,user_dir \
+    matrixdotorg/synapse:workers
+```
+
+The `SYNAPSE_WORKERS` environment variable is a comma-separated list of workers to use
+when running the container. All possible worker names are defined by the keys of the
+`WORKERS_CONFIG` variable in [this script](configure_workers_and_start.py), which the
+Dockerfile makes use of to generate appropriate worker, nginx and supervisord config files.
+
+Sharding is supported for a subset of workers, in line with the [worker documentation]
+(../docs/workers.md). To run multiple instances of a given worker type, simply specify 
+the type multiple times in `SYNAPSE_WORKERS`
+(e.g `SYNAPSE_WORKERS=event_creator,event_creator...`).
+
+Otherwise, `SYNAPSE_WORKERS` can either be left empty or unset to spawn no workers (
+leaving only the main process). The container is configured to use redis-based worker
+mode.
+
+Setting `SYNAPSE_WORKERS_WRITE_LOGS_TO_DISK=1` will cause worker logs to be written to 
+`<data_dir>/logs/<worker_name>.log`. Logs are kept for 1 week and rotate every day at 
+00:00, according to the container's clock.

docker/conf/homeserver.yaml

@@ -40,7 +40,9 @@ listeners:
         compress: false
   {% endif %}
 
-  - port: 8008
+  # Allow configuring in case we want to reverse proxy 8008
+  # using another process in the same container
+  - port: {{ SYNAPSE_HTTP_PORT or 8008 }}
     tls: false
     bind_addresses: ['::']
     type: http

docker/conf/log.config

@@ -2,9 +2,34 @@ version: 1
 
 formatters:
   precise:
-   format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s - %(message)s'
+{% if worker_name %}
+    format: '%(asctime)s - worker:{{ worker_name }} - %(name)s - %(lineno)d - %(levelname)s - %(request)s - %(message)s'
+{% else %}
+    format: '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s - %(message)s'
+{% endif %}
 
 handlers:
+  file:
+    class: logging.handlers.TimedRotatingFileHandler
+    formatter: precise
+    filename: {{ LOG_FILE_PATH or "homeserver.log" }}
+    when: "midnight"
+    backupCount: 6  # Does not include the current log file.
+    encoding: utf8
+
+  # Default to buffering writes to log file for efficiency. This means that
+  # will be a delay for INFO/DEBUG logs to get written, but WARNING/ERROR
+  # logs will still be flushed immediately.
+  buffer:
+    class: logging.handlers.MemoryHandler
+    target: file
+    # The capacity is the number of log lines that are buffered before
+    # being written to disk. Increasing this will lead to better
+    # performance, at the expensive of it taking longer for log lines to
+    # be written to disk.
+    capacity: 10
+    flushLevel: 30  # Flush for WARNING logs as well
+
   console:
     class: logging.StreamHandler
     formatter: precise
@@ -17,6 +42,12 @@ loggers:
 
 root:
     level: {{ SYNAPSE_LOG_LEVEL or "INFO" }}
+
+{% if LOG_FILE_PATH %}
+    handlers: [console, buffer]
+{% else %}
+    # Write logs to the console
     handlers: [console]
+{% endif %}
 
 disable_existing_loggers: false

docker/conf/nginx.conf.j2

@@ -0,0 +1,27 @@
+# This file contains the base config for the reverse proxy, as part of ../Dockerfile-workers.
+# configure_workers_and_start.py uses and amends to this file depending on the workers
+# that have been selected.
+
+{{ upstream_directives }}
+
+server {
+    # Listen on an unoccupied port number
+    listen 8008;
+    listen [::]:8008;
+
+    server_name localhost;
+
+    # Nginx by default only allows file uploads up to 1M in size
+    # Increase client_max_body_size to match max_upload_size defined in homeserver.yaml
+    client_max_body_size 100M;
+
+{{ worker_locations }}
+
+    # Send all other traffic to the main process
+    location ~* ^(\\/_matrix|\\/_synapse) {
+        proxy_pass http://localhost:8080;
+        proxy_set_header X-Forwarded-For $remote_addr;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_set_header Host $host;
+    }
+}

docker/conf/shared.yaml.j2

@@ -0,0 +1,9 @@
+# This file contains the base for the shared homeserver config file between Synapse workers,
+# as part of ./Dockerfile-workers.
+# configure_workers_and_start.py uses and amends to this file depending on the workers
+# that have been selected.
+
+redis:
+    enabled: true
+
+{{ shared_worker_config }}

docker/conf/supervisord.conf.j2

@@ -0,0 +1,41 @@
+# This file contains the base config for supervisord, as part of ../Dockerfile-workers.
+# configure_workers_and_start.py uses and amends to this file depending on the workers
+# that have been selected.
+[supervisord]
+nodaemon=true
+user=root
+
+[program:nginx]
+command=/usr/sbin/nginx -g "daemon off;"
+priority=500
+stdout_logfile=/dev/stdout
+stdout_logfile_maxbytes=0
+stderr_logfile=/dev/stderr
+stderr_logfile_maxbytes=0
+username=www-data
+autorestart=true
+
+[program:redis]
+command=/usr/bin/redis-server /etc/redis/redis.conf --daemonize no
+priority=1
+stdout_logfile=/dev/stdout
+stdout_logfile_maxbytes=0
+stderr_logfile=/dev/stderr
+stderr_logfile_maxbytes=0
+username=redis
+autorestart=true
+
+[program:synapse_main]
+command=/usr/local/bin/python -m synapse.app.homeserver --config-path="{{ main_config_path }}" --config-path=/conf/workers/shared.yaml
+priority=10
+# Log startup failures to supervisord's stdout/err
+# Regular synapse logs will still go in the configured data directory
+stdout_logfile=/dev/stdout
+stdout_logfile_maxbytes=0
+stderr_logfile=/dev/stderr
+stderr_logfile_maxbytes=0
+autorestart=unexpected
+exitcodes=0
+
+# Additional process blocks
+{{ worker_config }}

docker/conf/worker.yaml.j2

@@ -0,0 +1,26 @@
+# This is a configuration template for a single worker instance, and is
+# used by Dockerfile-workers.
+# Values will be change depending on whichever workers are selected when
+# running that image.
+
+worker_app: "{{ app }}"
+worker_name: "{{ name }}"
+
+# The replication listener on the main synapse process.
+worker_replication_host: 127.0.0.1
+worker_replication_http_port: 9093
+
+worker_listeners:
+  - type: http
+    port: {{ port }}
+{% if listener_resources %}
+    resources:
+      - names:
+{%- for resource in listener_resources %}
+        - {{ resource }}
+{%- endfor %}
+{% endif %}
+
+worker_log_config: {{ worker_log_config_filepath }}
+
+{{ worker_extra_conf }}