Merge branch 'develop' of github.com:matrix-org/synapse into matrix-o…

…rg-hotfixes
matrix-org · Feb 18, 2020 · bc936b5 · bc936b5
2 parents d6eae54 + 8a380d0
commit bc936b5
Show file tree

Hide file tree

Showing 121 changed files with 1,980 additions and 1,220 deletions.
diff --git a/.buildkite/worker-blacklist b/.buildkite/worker-blacklist
@@ -39,3 +39,5 @@ Server correctly handles incoming m.device_list_update
 
 # this fails reliably with a torture level of 100 due to https://github.com/matrix-org/synapse/issues/6536
 Outbound federation requests missing prev_events and then asks for /state_ids and resolves the state
+
+Can get rooms/{roomId}/members at a given point
diff --git a/CHANGES.md b/CHANGES.md
@@ -1,19 +1,43 @@
-Synapse 1.10.0rc4 (2020-02-11)
+Synapse 1.10.1 (2020-02-17)
+===========================
+
+Bugfixes
+--------
+
+- Fix a bug introduced in Synapse 1.10.0 which would cause room state to be cleared in the database if Synapse was upgraded direct from 1.2.1 or earlier to 1.10.0. ([\#6924](https://github.com/matrix-org/synapse/issues/6924))
+
+
+Synapse 1.10.0 (2020-02-12)
+===========================
+
+**WARNING to client developers**: As of this release Synapse validates `client_secret` parameters in the Client-Server API as per the spec. See [\#6766](https://github.com/matrix-org/synapse/issues/6766) for details.
+
+Updates to the Docker image
+---------------------------
+
+- Update the docker images to Alpine Linux 3.11. ([\#6897](https://github.com/matrix-org/synapse/issues/6897))
+
+
+Synapse 1.10.0rc5 (2020-02-11)
 ==============================
 
-Features
+Bugfixes
 --------
 
-- Filter out m.room.aliases from /sync state blocks until a full fix lands. ([\#6884](https://github.com/matrix-org/synapse/issues/6884))
+- Fix the filtering introduced in 1.10.0rc3 to also apply to the state blocks returned by `/sync`. ([\#6884](https://github.com/matrix-org/synapse/issues/6884))
 
+Synapse 1.10.0rc4 (2020-02-11)
+==============================
+
+This release candidate was built incorrectly and is superceded by 1.10.0rc5.
 
 Synapse 1.10.0rc3 (2020-02-10)
 ==============================
 
 Features
 --------
 
-- Filter out m.room.aliases from the CS API to mitigate abuse while a better solution is specced. ([\#6878](https://github.com/matrix-org/synapse/issues/6878))
+- Filter out `m.room.aliases` from the CS API to mitigate abuse while a better solution is specced. ([\#6878](https://github.com/matrix-org/synapse/issues/6878))
 
 
 Internal Changes
@@ -41,9 +65,6 @@ Internal Changes
 Synapse 1.10.0rc1 (2020-01-31)
 ==============================
 
-**WARNING to client developers**: As of this release Synapse validates `client_secret` parameters in the Client-Server API as per the spec. See [\#6766](https://github.com/matrix-org/synapse/issues/6766) for details.
-
-
 Features
 --------
 

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -200,6 +200,20 @@ Git allows you to add this signoff automatically when using the `-s`
 flag to `git commit`, which uses the name and email set in your
 `user.name` and `user.email` git configs.
 
+## Merge Strategy
+
+We use the commit history of develop/master extensively to identify
+when regressions were introduced and what changes have been made.
+
+We aim to have a clean merge history, which means we normally squash-merge
+changes into develop. For small changes this means there is no need to rebase
+to clean up your PR before merging. Larger changes with an organised set of
+commits may be merged as-is, if the history is judged to be useful.
+
+This use of squash-merging will mean PRs built on each other will be hard to 
+merge. We suggest avoiding these where possible, and if required, ensuring
+each PR has a tidy set of commits to ease merging.
+
 ## Conclusion
 
 That's it! Matrix is a very open and collaborative project as you might expect

diff --git a/INSTALL.md b/INSTALL.md
@@ -388,15 +388,17 @@ Once you have installed synapse as above, you will need to configure it.
 
 ## TLS certificates
 
-The default configuration exposes a single HTTP port: http://localhost:8008. It
-is suitable for local testing, but for any practical use, you will either need
-to enable a reverse proxy, or configure Synapse to expose an HTTPS port.
+The default configuration exposes a single HTTP port on the local
+interface: `http://localhost:8008`. It is suitable for local testing,
+but for any practical use, you will need Synapse's APIs to be served
+over HTTPS.
 
-For information on using a reverse proxy, see
+The recommended way to do so is to set up a reverse proxy on port
+`8448`. You can find documentation on doing so in
 [docs/reverse_proxy.md](docs/reverse_proxy.md).
 
-To configure Synapse to expose an HTTPS port, you will need to edit
-`homeserver.yaml`, as follows:
+Alternatively, you can configure Synapse to expose an HTTPS port. To do
+so, you will need to edit `homeserver.yaml`, as follows:
 
 * First, under the `listeners` section, uncomment the configuration for the
  TLS-enabled listener. (Remove the hash sign (`#`) at the start of
@@ -414,11 +416,15 @@ To configure Synapse to expose an HTTPS port, you will need to edit
  point these settings at an existing certificate and key, or you can
  enable Synapse's built-in ACME (Let's Encrypt) support. Instructions
  for having Synapse automatically provision and renew federation
- certificates through ACME can be found at [ACME.md](docs/ACME.md). If you
- are using your own certificate, be sure to use a `.pem` file that includes
- the full certificate chain including any intermediate certificates (for
- instance, if using certbot, use `fullchain.pem` as your certificate, not
- `cert.pem`).
+ certificates through ACME can be found at [ACME.md](docs/ACME.md).
+ Note that, as pointed out in that document, this feature will not
+ work with installs set up after November 2020. 
+
+ If you are using your
+ own certificate, be sure to use a `.pem` file that includes the full
+ certificate chain including any intermediate certificates (for
+ instance, if using certbot, use `fullchain.pem` as your certificate,
+ not `cert.pem`).
 
 For a more detailed guide to configuring your server for federation, see
 [federate.md](docs/federate.md)

diff --git a/README.rst b/README.rst
@@ -272,7 +272,7 @@ to install using pip and a virtualenv::
 
  virtualenv -p python3 env
  source env/bin/activate
- python -m pip install --no-use-pep517 -e .[all]
+ python -m pip install --no-use-pep517 -e ".[all]"
 
 This will run a process of downloading and installing all the needed
 dependencies into a virtual env.

diff --git a/changelog.d/6769.feature b/changelog.d/6769.feature
@@ -0,0 +1 @@
+Admin API to add or modify threepids of user accounts.
diff --git a/changelog.d/6781.bugfix b/changelog.d/6781.bugfix
@@ -0,0 +1 @@
+Fixed third party event rules function `on_create_room`'s return value being ignored.
diff --git a/changelog.d/6821.misc b/changelog.d/6821.misc
@@ -0,0 +1 @@
+Add type hints to `SyncHandler`.
diff --git a/changelog.d/6823.misc b/changelog.d/6823.misc
@@ -0,0 +1 @@
+Refactoring work in preparation for changing the event redaction algorithm.
diff --git a/changelog.d/6825.bugfix b/changelog.d/6825.bugfix
@@ -0,0 +1 @@
+Allow URL-encoded User IDs on `/_synapse/admin/v2/users/<user_id>[/admin]` endpoints. Thanks to @NHAS for reporting.
diff --git a/changelog.d/6827.misc b/changelog.d/6827.misc
@@ -0,0 +1 @@
+Refactoring work in preparation for changing the event redaction algorithm.
diff --git a/changelog.d/6833.misc b/changelog.d/6833.misc
@@ -0,0 +1 @@
+Reducing log level to DEBUG for synapse.storage.TIME.
diff --git a/changelog.d/6834.misc b/changelog.d/6834.misc
@@ -0,0 +1 @@
+Change the default power levels of invites, tombstones and server ACLs for new rooms.
diff --git a/changelog.d/6836.misc b/changelog.d/6836.misc
@@ -0,0 +1 @@
+Fix stacktraces when using `ObservableDeferred` and async/await.
diff --git a/changelog.d/6837.misc b/changelog.d/6837.misc
@@ -0,0 +1 @@
+Port much of `synapse.handlers.federation` to async/await.
diff --git a/changelog.d/6840.misc b/changelog.d/6840.misc
@@ -0,0 +1 @@
+Port much of `synapse.handlers.federation` to async/await.
diff --git a/changelog.d/6844.bugfix b/changelog.d/6844.bugfix
@@ -0,0 +1 @@
+Fix an issue with cross-signing where device signatures were not sent to remote servers.
diff --git a/changelog.d/6846.doc b/changelog.d/6846.doc
@@ -0,0 +1 @@
+Add details of PR merge strategy to contributing docs.
diff --git a/changelog.d/6847.misc b/changelog.d/6847.misc
@@ -0,0 +1 @@
+Populate `rooms.room_version` database column at startup, rather than in a background update.
diff --git a/changelog.d/6849.bugfix b/changelog.d/6849.bugfix
@@ -0,0 +1 @@
+Fix Synapse refusing to start if `federation_certificate_verification_whitelist` option is blank.
diff --git a/changelog.d/6854.misc b/changelog.d/6854.misc
@@ -0,0 +1 @@
+Refactoring work in preparation for changing the event redaction algorithm.
diff --git a/changelog.d/6855.misc b/changelog.d/6855.misc
@@ -0,0 +1 @@
+Update pip install directiosn in readme to avoid error when using zsh.
diff --git a/changelog.d/6856.misc b/changelog.d/6856.misc
@@ -0,0 +1 @@
+Refactoring work in preparation for changing the event redaction algorithm.
diff --git a/changelog.d/6857.misc b/changelog.d/6857.misc
@@ -0,0 +1 @@
+Refactoring work in preparation for changing the event redaction algorithm.
diff --git a/changelog.d/6858.misc b/changelog.d/6858.misc
@@ -0,0 +1 @@
+Refactoring work in preparation for changing the event redaction algorithm.
diff --git a/changelog.d/6862.misc b/changelog.d/6862.misc
@@ -0,0 +1 @@
+Reduce amount we log at `INFO` level.
diff --git a/changelog.d/6864.misc b/changelog.d/6864.misc
@@ -0,0 +1 @@
+Limit the number of events that can be requested by the backfill federation API to 100.
diff --git a/changelog.d/6869.misc b/changelog.d/6869.misc
@@ -0,0 +1 @@
+Remove unused `get_room_stats_state` method.
diff --git a/changelog.d/6871.misc b/changelog.d/6871.misc
@@ -0,0 +1 @@
+Add typing to `synapse.federation.sender` and port to async/await.
diff --git a/changelog.d/6877.removal b/changelog.d/6877.removal
@@ -0,0 +1 @@
+Remove `m.lazy_load_members` from `unstable_features` since lazy loading is in the stable Client-Server API version r0.5.0.
diff --git a/changelog.d/6882.misc b/changelog.d/6882.misc
@@ -0,0 +1 @@
+Reject device display names over 100 characters in length.
diff --git a/changelog.d/6883.misc b/changelog.d/6883.misc
@@ -0,0 +1 @@
+Add an additional entry to the SyTest blacklist for worker mode.
diff --git a/changelog.d/6887.misc b/changelog.d/6887.misc
@@ -0,0 +1 @@
+Fix the use of sed in the linting scripts when using BSD sed.
diff --git a/changelog.d/6888.feature b/changelog.d/6888.feature
@@ -0,0 +1 @@
+The result of a user directory search can now be filtered via the spam checker.
diff --git a/changelog.d/6891.doc b/changelog.d/6891.doc
@@ -0,0 +1 @@
+Spell out that the last event sent to a room won't be deleted by a purge. 
diff --git a/changelog.d/6901.misc b/changelog.d/6901.misc
@@ -0,0 +1 @@
+Return a 404 instead of 200 for querying information of a non-existant user through the admin API.
diff --git a/changelog.d/6904.removal b/changelog.d/6904.removal
@@ -0,0 +1 @@
+Stop sending alias events during adding / removing aliases. Check alt_aliases in the latest canonical aliases event when deleting an alias.
diff --git a/changelog.d/6905.doc b/changelog.d/6905.doc
@@ -0,0 +1 @@
+Update Synapse's documentation to warn about the deprecation of ACME v1.
diff --git a/changelog.d/6906.doc b/changelog.d/6906.doc
@@ -0,0 +1 @@
+Add documentation for the spam checker.
diff --git a/changelog.d/6909.doc b/changelog.d/6909.doc
@@ -0,0 +1 @@
+Update Synapse's documentation to warn about the deprecation of ACME v1.
diff --git a/changelog.d/6915.misc b/changelog.d/6915.misc
@@ -0,0 +1 @@
+Add type hints to the spam checker module.
diff --git a/changelog.d/6918.docker b/changelog.d/6918.docker
@@ -0,0 +1 @@
+The deprecated "generate-config-on-the-fly" mode is no longer supported.
diff --git a/changelog.d/6919.misc b/changelog.d/6919.misc
@@ -0,0 +1 @@
+Convert the directory handler tests to use HomeserverTestCase.
diff --git a/changelog.d/6920.misc b/changelog.d/6920.misc
@@ -0,0 +1 @@
+Add a warning about indentation to generated configuration files.
diff --git a/changelog.d/6921.docker b/changelog.d/6921.docker
@@ -0,0 +1 @@
+Databases created using the compose file in contrib/docker will now always have correct encoding and locale settings. Contributed by Fridtjof Mund.
diff --git a/changelog.d/6937.misc b/changelog.d/6937.misc
@@ -0,0 +1 @@
+Increase perf of `get_auth_chain_ids` used in state res v2.
diff --git a/changelog.d/6938.doc b/changelog.d/6938.doc
@@ -0,0 +1 @@
+Fix worker docs to point `/publicised_groups` API correctly.
diff --git a/contrib/docker/docker-compose.yml b/contrib/docker/docker-compose.yml
@@ -56,6 +56,9 @@ services:
  environment:
  - POSTGRES_USER=synapse
  - POSTGRES_PASSWORD=changeme
+ # ensure the database gets created correctly
+ # https://github.com/matrix-org/synapse/blob/master/docs/postgres.md#set-up-database
+ - POSTGRES_INITDB_ARGS="--encoding=UTF-8 --lc-collate=C --lc-ctype=C"
  volumes:
  # You may store the database tables in a local folder..
  - ./schemas:/var/lib/postgresql/data

diff --git a/debian/changelog b/debian/changelog
@@ -1,3 +1,15 @@
+matrix-synapse-py3 (1.10.1) stable; urgency=medium
+
+ * New synapse release 1.10.1.
+
+ -- Synapse Packaging team <packages@matrix.org> Mon, 17 Feb 2020 16:27:28 +0000
+
+matrix-synapse-py3 (1.10.0) stable; urgency=medium
+
+ * New synapse release 1.10.0.
+
+ -- Synapse Packaging team <packages@matrix.org> Wed, 12 Feb 2020 12:18:54 +0000
+
 matrix-synapse-py3 (1.9.1) stable; urgency=medium
 
  * New synapse release 1.9.1.

diff --git a/docker/Dockerfile b/docker/Dockerfile
@@ -16,7 +16,7 @@ ARG PYTHON_VERSION=3.7
 ###
 ### Stage 0: builder
 ###
-FROM docker.io/python:${PYTHON_VERSION}-alpine3.10 as builder
+FROM docker.io/python:${PYTHON_VERSION}-alpine3.11 as builder
 
 # install the OS build deps
 

diff --git a/docker/README.md b/docker/README.md
@@ -110,12 +110,12 @@ argument to `docker run`.
 
 ## Legacy dynamic configuration file support
 
-For backwards-compatibility only, the docker image supports creating a dynamic
-configuration file based on environment variables. This is now deprecated, but
-is enabled when the `SYNAPSE_SERVER_NAME` variable is set (and `generate` is
-not given).
+The docker image used to support creating a dynamic configuration file based
+on environment variables. This is no longer supported, and an error will be
+raised if you try to run synapse without a config file.
 
-To migrate from a dynamic configuration file to a static one, run the docker
+It is, however, possible to generate a static configuration file based on
+the environment variables that were previously used. To do this, run the docker
 container once with the environment variables set, and `migrate_config`
 command line option. For example:
 
@@ -127,15 +127,20 @@ docker run -it --rm \
  matrixdotorg/synapse:latest migrate_config
 ```
 
-This will generate the same configuration file as the legacy mode used, but
-will store it in `/data/homeserver.yaml` instead of a temporary location. You
-can then use it as shown above at [Running synapse](#running-synapse).
+This will generate the same configuration file as the legacy mode used, and
+will store it in `/data/homeserver.yaml`. You can then use it as shown above at
+[Running synapse](#running-synapse).
+
+Note that the defaults used in this configuration file may be different to
+those when generating a new config file with `generate`: for example, TLS is
+enabled by default in this mode. You are encouraged to inspect the generated
+configuration file and edit it to ensure it meets your needs.
 
 ## Building the image
 
 If you need to build the image from a Synapse checkout, use the following `docker
  build` command from the repo's root:
- 
+
 ```
 docker build -t matrixdotorg/synapse -f docker/Dockerfile .
 ```

diff --git a/docker/start.py b/docker/start.py
@@ -188,11 +188,6 @@ def main(args, environ):
  else:
  ownership = "{}:{}".format(desired_uid, desired_gid)
 
- log(
- "Container running as UserID %s:%s, ENV (or defaults) requests %s:%s"
- % (os.getuid(), os.getgid(), desired_uid, desired_gid)
- )
-
  if ownership is None:
  log("Will not perform chmod/su-exec as UserID already matches request")
 
@@ -213,38 +208,30 @@ def main(args, environ):
  if mode is not None:
  error("Unknown execution mode '%s'" % (mode,))
 
- if "SYNAPSE_SERVER_NAME" in environ:
- # backwards-compatibility generate-a-config-on-the-fly mode
- if "SYNAPSE_CONFIG_PATH" in environ:
+ config_dir = environ.get("SYNAPSE_CONFIG_DIR", "/data")
+ config_path = environ.get("SYNAPSE_CONFIG_PATH", config_dir + "/homeserver.yaml")
+
+ if not os.path.exists(config_path):
+ if "SYNAPSE_SERVER_NAME" in environ:
  error(
- "SYNAPSE_SERVER_NAME can only be combined with SYNAPSE_CONFIG_PATH "
- "in `generate` or `migrate_config` mode. To start synapse using a "
- "config file, unset the SYNAPSE_SERVER_NAME environment variable."
+ """\
+Config file '%s' does not exist.
+
+The synapse docker image no longer supports generating a config file on-the-fly
+based on environment variables. You can migrate to a static config file by
+running with 'migrate_config'. See the README for more details.
+"""
+ % (config_path,)
  )
 
- config_path = "/compiled/homeserver.yaml"
- log(
- "Generating config file '%s' on-the-fly from environment variables.\n"
- "Note that this mode is deprecated. You can migrate to a static config\n"
- "file by running with 'migrate_config'. See the README for more details."
+ error(
+  "Config file '%s' does not exist. You should either create a new "
+ "config file by running with the `generate` argument (and then edit "
+ "the resulting file before restarting) or specify the path to an "
+ "existing config file with the SYNAPSE_CONFIG_PATH variable."
  % (config_path,)
  )
 
- generate_config_from_template("/compiled", config_path, environ, ownership)
- else:
- config_dir = environ.get("SYNAPSE_CONFIG_DIR", "/data")
- config_path = environ.get(
- "SYNAPSE_CONFIG_PATH", config_dir + "/homeserver.yaml"
- )
- if not os.path.exists(config_path):
- error(
- "Config file '%s' does not exist. You should either create a new "
- "config file by running with the `generate` argument (and then edit "
- "the resulting file before restarting) or specify the path to an "
- "existing config file with the SYNAPSE_CONFIG_PATH variable."
- % (config_path,)
- )
-
  log("Starting synapse with config file " + config_path)
 
  args = ["python", "-m", synapse_worker, "--config-path", config_path]

diff --git a/docs/.sample_config_header.yaml b/docs/.sample_config_header.yaml
@@ -1,4 +1,4 @@
-# The config is maintained as an up-to-date snapshot of the default
+# This file is maintained as an up-to-date snapshot of the default
 # homeserver.yaml configuration generated by Synapse.
 #
 # It is intended to act as a reference for the default configuration,
@@ -10,3 +10,5 @@
 # homeserver.yaml. Instead, if you are starting from scratch, please generate
 # a fresh config using Synapse by following the instructions in INSTALL.md.
 
+################################################################################
+