Merge remote-tracking branch 'origin/develop' into shhs

matrix-org · Jul 2, 2019 · 4689408 · 4689408
2 parents bed45ab + b491468
commit 4689408
Show file tree

Hide file tree

Showing 461 changed files with 17,745 additions and 11,322 deletions.
diff --git a/.buildkite/pipeline.yml b/.buildkite/pipeline.yml
@@ -5,8 +5,8 @@ steps:
 
   - command:
       - "python -m pip install tox"
-      - "tox -e pep8"
-    label: "\U0001F9F9 PEP-8"
+      - "tox -e check_codestyle"
+    label: "\U0001F9F9 Check Style"
     plugins:
       - docker#v3.0.1:
           image: "python:3.6"

diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -4,7 +4,7 @@ jobs:
     machine: true
     steps:
       - checkout
-      - run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:${CIRCLE_TAG} -t matrixdotorg/synapse:${CIRCLE_TAG}-py3 --build-arg PYTHON_VERSION=3.6 .
+      - run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:${CIRCLE_TAG} -t matrixdotorg/synapse:${CIRCLE_TAG}-py3 .
       - run: docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
       - run: docker push matrixdotorg/synapse:${CIRCLE_TAG}
       - run: docker push matrixdotorg/synapse:${CIRCLE_TAG}-py3
@@ -19,7 +19,7 @@ jobs:
     machine: true
     steps:
       - checkout
-      - run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:latest -t matrixdotorg/synapse:latest-py3 --build-arg PYTHON_VERSION=3.6 .
+      - run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:latest -t matrixdotorg/synapse:latest-py3 .
       - run: docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
       - run: docker push matrixdotorg/synapse:latest
       - run: docker push matrixdotorg/synapse:latest-py3

diff --git a/.dockerignore b/.dockerignore
@@ -1,9 +1,13 @@
-Dockerfile
-.travis.yml
-.gitignore
-demo/etc
-tox.ini
-.git/*
-.tox/*
-debian/matrix-synapse/
-debian/matrix-synapse-*/
+# ignore everything by default
+*
+
+# things to include
+!docker
+!scripts
+!synapse
+!MANIFEST.in
+!README.rst
+!setup.py
+!synctl
+
+**/__pycache__
diff --git a/AUTHORS.rst b/AUTHORS.rst
@@ -72,3 +72,6 @@ Jason Robinson <jasonr at matrix.org>
 
 Joseph Weston <joseph at weston.cloud>
  + Add admin API for querying HS version
+
+Benjamin Saunders <ben.e.saunders at gmail dot com>
+ * Documentation improvements
diff --git a/CHANGES.md b/CHANGES.md
@@ -52,7 +52,11 @@ Features
 - Add a script to generate new signing-key files. ([\#5361](https://github.com/matrix-org/synapse/issues/5361))
 - Update upgrade and installation guides ahead of 1.0. ([\#5371](https://github.com/matrix-org/synapse/issues/5371))
 - Replace the `perspectives` configuration section with `trusted_key_servers`, and make validating the signatures on responses optional (since TLS will do this job for us). ([\#5374](https://github.com/matrix-org/synapse/issues/5374))
-- Add ability to perform password reset via email without trusting the identity server. ([\#5377](https://github.com/matrix-org/synapse/issues/5377))
+- Add ability to perform password reset via email without trusting the identity server. **As a result of this PR, password resets will now be disabled on the default configuration.**
+
+  Password reset emails are now sent from the homeserver by default, instead of the identity server. To enable this functionality, ensure `email` and `public_baseurl` config options are filled out.
+
+  If you would like to re-enable password resets being sent from the identity server (warning: this is dangerous! See [#5345](https://github.com/matrix-org/synapse/pull/5345)), set `email.trust_identity_server_for_password_resets` to true. ([\#5377](https://github.com/matrix-org/synapse/issues/5377))
 - Set default room version to v4. ([\#5379](https://github.com/matrix-org/synapse/issues/5379))
 
 

diff --git a/INSTALL.md b/INSTALL.md
@@ -1,3 +1,4 @@
+- [Choosing your server name](#choosing-your-server-name)
 - [Installing Synapse](#installing-synapse)
   - [Installing from source](#installing-from-source)
     - [Platform-Specific Instructions](#platform-specific-instructions)
@@ -10,6 +11,22 @@
   - [Setting up a TURN server](#setting-up-a-turn-server)
   - [URL previews](#url-previews)
 
+# Choosing your server name
+
+It is important to choose the name for your server before you install Synapse,
+because it cannot be changed later.
+
+The server name determines the "domain" part of user-ids for users on your
+server: these will all be of the format `@user:my.domain.name`. It also
+determines how other matrix servers will reach yours for federation.
+
+For a test configuration, set this to the hostname of your server. For a more
+production-ready setup, you will probably want to specify your domain
+(`example.com`) rather than a matrix-specific hostname here (in the same way
+that your email address is probably `user@example.com` rather than
+`user@email.example.com`) - but doing so may require more advanced setup: see
+[Setting up Federation](docs/federate.md).
+
 # Installing Synapse
 
 ## Installing from source
@@ -64,16 +81,7 @@ python -m synapse.app.homeserver \
     --report-stats=[yes|no]
 ```
 
-... substituting an appropriate value for `--server-name`. The server name
-determines the "domain" part of user-ids for users on your server: these will
-all be of the format `@user:my.domain.name`. It also determines how other
-matrix servers will reach yours for Federation. For a test configuration,
-set this to the hostname of your server. For a more production-ready setup, you
-will probably want to specify your domain (`example.com`) rather than a
-matrix-specific hostname here (in the same way that your email address is
-probably `user@example.com` rather than `user@email.example.com`) - but
-doing so may require more advanced setup: see [Setting up Federation](docs/federate.md).
-Beware that the server name cannot be changed later.
+... substituting an appropriate value for `--server-name`.
 
 This command will generate you a config file that you can then customise, but it will
 also generate a set of keys for you. These keys will allow your Home Server to
@@ -86,9 +94,6 @@ different. See the
 [spec](https://matrix.org/docs/spec/server_server/latest.html#retrieving-server-keys)
 for more information on key management.)
 
-You will need to give Synapse a TLS certficate before it will start - see [TLS
-certificates](#tls-certificates).
-
 To actually run your new homeserver, pick a working directory for Synapse to
 run (e.g. `~/synapse`), and::
 

diff --git a/README.rst b/README.rst
@@ -340,8 +340,11 @@ log lines and looking for any 'Processed request' lines which take more than
 a few seconds to execute. Please let us know at #synapse:matrix.org if
 you see this failure mode so we can help debug it, however.
 
-Help!! Synapse eats all my RAM!
--------------------------------
+Help!! Synapse is slow and eats all my RAM/CPU!
+-----------------------------------------------
+
+First, ensure you are running the latest version of Synapse, using Python 3
+with a PostgreSQL database.
 
 Synapse's architecture is quite RAM hungry currently - we deliberately
 cache a lot of recent room data and metadata in RAM in order to speed up
@@ -352,14 +355,29 @@ variable. The default is 0.5, which can be decreased to reduce RAM usage
 in memory constrained enviroments, or increased if performance starts to
 degrade.
 
+However, degraded performance due to a low cache factor, common on
+machines with slow disks, often leads to explosions in memory use due
+backlogged requests. In this case, reducing the cache factor will make
+things worse. Instead, try increasing it drastically. 2.0 is a good
+starting value.
+
 Using `libjemalloc <http://jemalloc.net/>`_ can also yield a significant
-improvement in overall amount, and especially in terms of giving back RAM
-to the OS. To use it, the library must simply be put in the LD_PRELOAD
-environment variable when launching Synapse. On Debian, this can be done
-by installing the ``libjemalloc1`` package and adding this line to
-``/etc/default/matrix-synapse``::
+improvement in overall memory use, and especially in terms of giving back
+RAM to the OS. To use it, the library must simply be put in the
+LD_PRELOAD environment variable when launching Synapse. On Debian, this
+can be done by installing the ``libjemalloc1`` package and adding this
+line to ``/etc/default/matrix-synapse``::
 
     LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1
 
 This can make a significant difference on Python 2.7 - it's unclear how
 much of an improvement it provides on Python 3.x.
+
+If you're encountering high CPU use by the Synapse process itself, you
+may be affected by a bug with presence tracking that leads to a
+massive excess of outgoing federation requests (see `discussion
+<https://github.com/matrix-org/synapse/issues/3971>`_). If metrics
+indicate that your server is also issuing far more outgoing federation
+requests than can be accounted for by your users' activity, this is a
+likely cause. The misbehavior can be worked around by setting
+``use_presence: false`` in the Synapse config file.
diff --git a/changelog.d/4276.misc b/changelog.d/4276.misc
@@ -0,0 +1 @@
+Improve README section on performance troubleshooting.
diff --git a/changelog.d/5015.misc b/changelog.d/5015.misc
@@ -0,0 +1 @@
+Add logging to 3pid invite signature verification.
diff --git a/changelog.d/5042.bugfix b/changelog.d/5042.bugfix
@@ -0,0 +1 @@
+Fix bug processing incoming events over federation if call to `/get_missing_events` fails.
diff --git a/changelog.d/5051.bugfix b/changelog.d/5051.bugfix
@@ -0,0 +1 @@
+Prevent >1 room upgrades happening simultaneously on the same room.
diff --git a/changelog.d/5092.feature b/changelog.d/5092.feature
@@ -0,0 +1 @@
+Added possibilty to disable local password authentication. Contributed by Daniel Hoffend.
diff --git a/changelog.d/5313.misc b/changelog.d/5313.misc
@@ -0,0 +1 @@
+Update example haproxy config to a more compatible setup.
diff --git a/changelog.d/5475.misc b/changelog.d/5475.misc
@@ -0,0 +1 @@
+Synapse can now handle RestServlets that return coroutines.
diff --git a/changelog.d/5476.misc b/changelog.d/5476.misc
@@ -0,0 +1 @@
+Add new metrics for number of forward extremities being persisted and number of state groups involved in resolution.
diff --git a/changelog.d/5480.misc b/changelog.d/5480.misc
@@ -0,0 +1 @@
+Add an EXPERIMENTAL config option to try and periodically clean up extremities by sending dummy events.
diff --git a/changelog.d/5482.misc b/changelog.d/5482.misc
@@ -0,0 +1 @@
+Synapse's codebase is now formatted by `black`.
diff --git a/changelog.d/5490.bugfix b/changelog.d/5490.bugfix
@@ -0,0 +1 @@
+Fix failure to start under docker with SAML support enabled.
diff --git a/changelog.d/5493.misc b/changelog.d/5493.misc
@@ -0,0 +1 @@
+Track deactivated accounts in the database.
diff --git a/changelog.d/5498.bugfix b/changelog.d/5498.bugfix
@@ -0,0 +1 @@
+Fix intermittent exceptions on Apple hardware. Also fix bug that caused database activity times to be under-reported in log lines.
diff --git a/changelog.d/5499.misc b/changelog.d/5499.misc
@@ -0,0 +1 @@
+Some cleanups and sanity-checking in the CPU and database metrics.
diff --git a/changelog.d/5500.bugfix b/changelog.d/5500.bugfix
@@ -0,0 +1 @@
+Fix logging error when a tampered event is detected.
diff --git a/changelog.d/5502.misc b/changelog.d/5502.misc
@@ -0,0 +1 @@
+Improve email notification logging.
diff --git a/changelog.d/5505.feature b/changelog.d/5505.feature
@@ -0,0 +1 @@
+Add support for handling pagination APIs on client reader worker.
diff --git a/changelog.d/5507.bugfix b/changelog.d/5507.bugfix
@@ -0,0 +1 @@
+Fix bug where clients could tight loop calling `/sync` for a period.
diff --git a/changelog.d/5509.misc b/changelog.d/5509.misc
@@ -0,0 +1 @@
+Fix "Unexpected entry in 'full_schemas'" log warning.
diff --git a/changelog.d/5510.misc b/changelog.d/5510.misc
@@ -0,0 +1 @@
+Improve logging when generating config files.
diff --git a/changelog.d/5511.misc b/changelog.d/5511.misc
@@ -0,0 +1 @@
+Refactor and clean up Config parser for maintainability.
diff --git a/changelog.d/5512.feature b/changelog.d/5512.feature
@@ -0,0 +1 @@
+Improve help and cmdline option names for --generate-config options.
diff --git a/changelog.d/5513.feature b/changelog.d/5513.feature
@@ -0,0 +1 @@
+Add support for handling pagination APIs on client reader worker.
diff --git a/changelog.d/5514.bugfix b/changelog.d/5514.bugfix
@@ -0,0 +1 @@
+Fix bug with `jinja2` preventing Synapse from starting. Users who had this problem should now simply need to run `pip install matrix-synapse`.
diff --git a/changelog.d/5516.feature b/changelog.d/5516.feature
@@ -0,0 +1 @@
+Allow configuration of the path used for ACME account keys.
diff --git a/changelog.d/5521.feature b/changelog.d/5521.feature
@@ -0,0 +1 @@
+Allow configuration of the path used for ACME account keys.
diff --git a/changelog.d/5522.feature b/changelog.d/5522.feature
@@ -0,0 +1 @@
+Allow configuration of the path used for ACME account keys.
diff --git a/changelog.d/5523.bugfix b/changelog.d/5523.bugfix
@@ -0,0 +1 @@
+Fix a regression where homeservers on private IP addresses were incorrectly blacklisted.
diff --git a/changelog.d/5524.feature b/changelog.d/5524.feature
@@ -0,0 +1 @@
+Add --data-dir and --open-private-ports options.
diff --git a/changelog.d/5525.removal b/changelog.d/5525.removal
@@ -0,0 +1 @@
+Remove support for cpu_affinity setting.
diff --git a/changelog.d/5531.feature b/changelog.d/5531.feature
@@ -0,0 +1 @@
+Add support for handling pagination APIs on client reader worker.
diff --git a/changelog.d/5534.feature b/changelog.d/5534.feature
@@ -0,0 +1 @@
+Split public rooms directory auth config in two settings, in order to manage client auth independently from the federation part of it. Obsoletes the "restrict_public_rooms_to_local_users" configuration setting. If "restrict_public_rooms_to_local_users" is set in the config, Synapse will act as if both new options are enabled, i.e. require authentication through the client API and deny federation requests.
diff --git a/changelog.d/5537.misc b/changelog.d/5537.misc
@@ -0,0 +1 @@
+Add information about how to install and run `black` on the codebase to code_style.rst.
diff --git a/changelog.d/5543.misc b/changelog.d/5543.misc
@@ -0,0 +1 @@
+Make the config clearer in that email.template_dir is relative to the Synapse's root directory, not the `synapse/` folder within it.
diff --git a/changelog.d/5545.misc b/changelog.d/5545.misc
@@ -0,0 +1 @@
+Update v1.0.0 release changelog to include more information about changes to password resets.
diff --git a/changelog.d/5546.feature b/changelog.d/5546.feature
@@ -0,0 +1 @@
+Update docker image to use Python 3.7.
diff --git a/changelog.d/5547.feature b/changelog.d/5547.feature
@@ -0,0 +1 @@
+Increase default log level for docker image to INFO. It can still be changed by editing the generated log.config file.
diff --git a/changelog.d/5548.misc b/changelog.d/5548.misc
@@ -0,0 +1 @@
+Remove non-functioning check_event_hash.py dev script.
diff --git a/changelog.d/5550.feature b/changelog.d/5550.feature
@@ -0,0 +1 @@
+The minimum TLS version used for outgoing federation requests can now be set with `federation_client_minimum_tls_version`.
diff --git a/changelog.d/5550.misc b/changelog.d/5550.misc
@@ -0,0 +1 @@
+Synapse will now only allow TLS v1.2 connections when serving federation, if it terminates TLS. As Synapse's allowed ciphers were only able to be used in TLSv1.2 before, this does not change behaviour.
diff --git a/changelog.d/5555.bugfix b/changelog.d/5555.bugfix
@@ -0,0 +1 @@
+Fixed m.login.jwt using unregistred user_id and added pyjwt>=1.6.4 as jwt conditional dependencies. Contributed by Pau Rodriguez-Estivill.
diff --git a/changelog.d/5557.misc b/changelog.d/5557.misc
@@ -0,0 +1 @@
+Logging when running GC collection on generation 0 is now at the DEBUG level, not INFO.
diff --git a/changelog.d/5558.misc b/changelog.d/5558.misc
@@ -0,0 +1 @@
+Improve install docs on choosing server_name.
diff --git a/changelog.d/5559.feature b/changelog.d/5559.feature
@@ -0,0 +1 @@
+Optimise devices changed query to not pull unnecessary rows from the database, reducing database load.
diff --git a/changelog.d/5561.feature b/changelog.d/5561.feature
@@ -0,0 +1 @@
+Update Docker image to deprecate the use of environment variables for configuration, and make the use of a static configuration the default.
diff --git a/changelog.d/5562.feature b/changelog.d/5562.feature
@@ -0,0 +1 @@
+Update Docker image to deprecate the use of environment variables for configuration, and make the use of a static configuration the default.
diff --git a/changelog.d/5563.bugfix b/changelog.d/5563.bugfix
@@ -0,0 +1 @@
+Docker: Use a sensible location for data files when generating a config file.
diff --git a/changelog.d/5564.misc b/changelog.d/5564.misc
@@ -0,0 +1 @@
+Reduce the amount of stuff we send in the docker context.
diff --git a/changelog.d/5565.feature b/changelog.d/5565.feature
@@ -0,0 +1 @@
+Docker: Send synapse logs to the docker logging system, by default.
diff --git a/changelog.d/5566.feature b/changelog.d/5566.feature
@@ -0,0 +1 @@
+Update Docker image to deprecate the use of environment variables for configuration, and make the use of a static configuration the default.
diff --git a/changelog.d/5567.feature b/changelog.d/5567.feature
@@ -0,0 +1 @@
+Update Docker image to deprecate the use of environment variables for configuration, and make the use of a static configuration the default.
diff --git a/changelog.d/5568.feature b/changelog.d/5568.feature
@@ -0,0 +1 @@
+Docker image: open the non-TLS port by default.
diff --git a/changelog.d/5570.misc b/changelog.d/5570.misc
@@ -0,0 +1 @@
+Point the reverse links in the Purge History contrib scripts at the intended location.
diff --git a/changelog.d/5576.bugfix b/changelog.d/5576.bugfix
@@ -0,0 +1 @@
+Fix a bug that would cause invited users to receive several emails for a single 3PID invite in case the inviter is rate limited.
diff --git a/changelog.d/5585.misc b/changelog.d/5585.misc
@@ -0,0 +1 @@
+Synapse can now handle RestServlets that return coroutines.