Merge remote-tracking branch 'upstream/develop' into room_admin

MANIFEST.in

@@ -30,23 +30,24 @@ recursive-include synapse/static *.gif
 recursive-include synapse/static *.html
 recursive-include synapse/static *.js
 
-exclude Dockerfile
+exclude .codecov.yml
+exclude .coveragerc
 exclude .dockerignore
-exclude test_postgresql.sh
 exclude .editorconfig
+exclude Dockerfile
+exclude mypy.ini
 exclude sytest-blacklist
+exclude test_postgresql.sh
 
 include pyproject.toml
 recursive-include changelog.d *
 
 prune .buildkite
 prune .circleci
-prune .codecov.yml
-prune .coveragerc
 prune .github
+prune contrib
 prune debian
 prune demo/etc
 prune docker
-prune mypy.ini
 prune snap
 prune stubs

UPGRADE.rst

@@ -75,6 +75,37 @@ for example:
      wget https://packages.matrix.org/debian/pool/main/m/matrix-synapse-py3/matrix-synapse-py3_1.3.0+stretch1_amd64.deb
      dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
 
+Upgrading to v1.13.0
+====================
+
+Incorrect database migration in old synapse versions
+----------------------------------------------------
+
+A bug was introduced in Synapse 1.4.0 which could cause the room directory to
+be incomplete or empty if Synapse was upgraded directly from v1.2.1 or earlier,
+to versions between v1.4.0 and v1.12.x.
+
+This will *not* be a problem for Synapse installations which were:
+ * created at v1.4.0 or later,
+ * upgraded via v1.3.x, or
+ * upgraded straight from v1.2.1 or earlier to v1.13.0 or later.
+
+If completeness of the room directory is a concern, installations which are
+affected can be repaired as follows:
+
+1. Run the following sql from a `psql` or `sqlite3` console:
+
+   .. code:: sql
+
+     INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
+        ('populate_stats_process_rooms', '{}', 'current_state_events_membership');
+
+     INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
+        ('populate_stats_process_users', '{}', 'populate_stats_process_rooms');
+
+2. Restart synapse.
+
+
 Upgrading to v1.12.0
 ====================
 

changelog.d/6881.misc

@@ -0,0 +1 @@
+Return total number of users and profile attributes in admin users endpoint. Contributed by Awesome Technologies Innovationslabor GmbH.

changelog.d/7091.doc

@@ -0,0 +1 @@
+Improve the documentation of application service configuration files.

changelog.d/7146.misc

@@ -0,0 +1 @@
+Run replication streamers on workers.

changelog.d/7172.misc

@@ -0,0 +1 @@
+Use `stream.current_token()` and remove `stream_positions()`.

changelog.d/7219.misc

@@ -1 +1 @@
-Add typing information to federation server code.
+Add typing annotations in `synapse.federation`.

changelog.d/7278.misc

@@ -0,0 +1 @@
+Add some unit tests for replication.

changelog.d/7281.misc

@@ -0,0 +1 @@
+Add MultiWriterIdGenerator to support multiple concurrent writers of streams.

changelog.d/7302.bugfix

@@ -0,0 +1 @@
+Persist user interactive authentication sessions across workers and Synapse restarts.

changelog.d/7316.bugfix

@@ -0,0 +1 @@
+Fixed backwards compatibility logic of the first value of `trusted_third_party_id_servers` being used for `account_threepid_delegates.email`, which occurs when the former, deprecated option is set and the latter is not.

changelog.d/7337.bugfix

@@ -0,0 +1 @@
+Fix a bug where event updates might not be sent over replication to worker processes after the stream falls behind.

changelog.d/7338.misc

@@ -0,0 +1 @@
+Convert some federation handler code to async/await.

changelog.d/7341.bugfix

@@ -0,0 +1 @@
+Fix bad error handling that would cause Synapse to crash if it's provided with a YAML configuration file that's either empty or doesn't parse into a key-value map.

changelog.d/7343.feature

@@ -0,0 +1 @@
+Support SSO in the user interactive authentication workflow.

changelog.d/7344.bugfix

@@ -0,0 +1 @@
+Fix incorrect metrics reporting for `renew_attestations` background task.

changelog.d/7352.feature

@@ -0,0 +1 @@
+Add support for running replication over Redis when using workers.

changelog.d/7357.doc

@@ -0,0 +1 @@
+Add documentation on monitoring workers with Prometheus.

changelog.d/7358.bugfix

@@ -0,0 +1 @@
+Fix a bug where event updates might not be sent over replication to worker processes after the stream falls behind.

changelog.d/7359.misc

@@ -0,0 +1 @@
+Fix collation for postgres for unit tests.

changelog.d/7361.doc

@@ -0,0 +1 @@
+Clarify endpoint usage in the users admin api documentation.

changelog.d/7363.misc

@@ -0,0 +1 @@
+Convert RegistrationWorkerStore.is_server_admin and dependent code to async/await.

changelog.d/7364.misc

@@ -0,0 +1 @@
+Add an `instance_name` to `RDATA` and `POSITION` replication commands.

changelog.d/7367.bugfix

@@ -0,0 +1 @@
+Prevent non-federating rooms from appearing in responses to federated `POST /publicRoom` requests when a filter was included.

changelog.d/7368.bugfix

@@ -0,0 +1 @@
+Improve error responses when accessing remote public room lists.

changelog.d/7369.misc

@@ -0,0 +1 @@
+Thread through instance name to replication client.

changelog.d/7374.misc

@@ -0,0 +1 @@
+Move catchup of replication streams logic to worker.

changelog.d/7378.misc

@@ -0,0 +1 @@
+Move catchup of replication streams logic to worker.

changelog.d/7382.misc

@@ -0,0 +1 @@
+Add typing annotations in `synapse.federation`.

changelog.d/7387.bugfix

@@ -0,0 +1 @@
+Fix a bug which would cause the room durectory to be incorrectly populated if Synapse was upgraded directly from v1.2.1 or earlier to v1.4.0 or later. Note that this fix does not apply retrospectively; see the [upgrade notes](UPGRADE.rst#upgrading-to-v1130) for more information.

changelog.d/7393.bugfix

@@ -0,0 +1 @@
+Fix bug in `EventContext.deserialize`.

changelog.d/7394.misc

@@ -0,0 +1 @@
+Convert synapse.server_notices to async/await.

changelog.d/7395.misc

@@ -0,0 +1 @@
+Convert synapse.notifier to async/await.

changelog.d/7396.misc

@@ -0,0 +1 @@
+Convert the room handler to async/await.

changelog.d/7401.feature

@@ -0,0 +1 @@
+Add support for running replication over Redis when using workers.

changelog.d/7404.misc

@@ -0,0 +1 @@
+Fix issues with the Python package manifest.

changelog.d/7408.misc

@@ -0,0 +1 @@
+Clean up some LoggingContext code.

changelog.d/7420.misc

@@ -0,0 +1 @@
+Prevent methods in `synapse.handlers.auth` from polling the homeserver config every request.

changelog.d/7421.misc

@@ -0,0 +1 @@
+Move catchup of replication streams logic to worker.

changelog.d/7423.misc

@@ -0,0 +1 @@
+Speed up fetching device lists changes when handling `/sync` requests.

changelog.d/7427.feature

@@ -0,0 +1 @@
+Add support for running replication over Redis when using workers.

changelog.d/7428.misc

@@ -0,0 +1 @@
+Improve performance of `get_e2e_cross_signing_key`.

changelog.d/7429.misc

@@ -0,0 +1 @@
+Improve performance of `mark_as_sent_devices_by_remote`.

docs/admin_api/user_admin_api.rst

@@ -33,12 +33,22 @@ with a body of:
 
 including an ``access_token`` of a server admin.
 
-The parameter ``displayname`` is optional and defaults to ``user_id``.
-The parameter ``threepids`` is optional.
-The parameter ``avatar_url`` is optional.
-The parameter ``admin`` is optional and defaults to 'false'.
-The parameter ``deactivated`` is optional and defaults to 'false'.
-The parameter ``password`` is optional. If provided the user's password is updated and all devices are logged out.
+The parameter ``displayname`` is optional and defaults to the value of
+``user_id``.
+
+The parameter ``threepids`` is optional and allows setting the third-party IDs
+(email, msisdn) belonging to a user.
+
+The parameter ``avatar_url`` is optional. Must be a [MXC
+URI](https://matrix.org/docs/spec/client_server/r0.6.0#matrix-content-mxc-uris).
+
+The parameter ``admin`` is optional and defaults to ``false``.
+
+The parameter ``deactivated`` is optional and defaults to ``false``.
+
+The parameter ``password`` is optional. If provided, the user's password is
+updated and all devices are logged out.
+
 If the user already exists then optional parameters default to the current value.
 
 List Accounts
@@ -51,16 +61,25 @@ The api is::
     GET /_synapse/admin/v2/users?from=0&limit=10&guests=false
 
 including an ``access_token`` of a server admin.
-The parameters ``from`` and ``limit`` are required only for pagination.
-By default, a ``limit`` of 100 is used.
-The parameter ``user_id`` can be used to select only users with user ids that
-contain this value.
-The parameter ``guests=false`` can be used to exclude guest users,
-default is to include guest users.
-The parameter ``deactivated=true`` can be used to include deactivated users,
-default is to exclude deactivated users.
-If the endpoint does not return a ``next_token`` then there are no more users left.
-It returns a JSON body like the following:
+
+The parameter ``from`` is optional but used for pagination, denoting the
+offset in the returned results. This should be treated as an opaque value and
+not explicitly set to anything other than the return value of ``next_token``
+from a previous call.
+
+The parameter ``limit`` is optional but is used for pagination, denoting the
+maximum number of items to return in this call. Defaults to ``100``.
+
+The parameter ``user_id`` is optional and filters to only users with user IDs
+that contain this value.
+
+The parameter ``guests`` is optional and if ``false`` will **exclude** guest users.
+Defaults to ``true`` to include guest users.
+
+The parameter ``deactivated`` is optional and if ``true`` will **include** deactivated users.
+Defaults to ``false`` to exclude deactivated users.
+
+A JSON body is returned with the following shape:
 
 .. code:: json
 
@@ -72,19 +91,29 @@ It returns a JSON body like the following:
                 "is_guest": 0,
                 "admin": 0,
                 "user_type": null,
-                "deactivated": 0
+                "deactivated": 0,
+                "displayname": "<User One>",
+                "avatar_url": null
             }, {
                 "name": "<user_id2>",
                 "password_hash": "<password_hash2>",
                 "is_guest": 0,
                 "admin": 1,
                 "user_type": null,
-                "deactivated": 0
+                "deactivated": 0,
+                "displayname": "<User Two>",
+                "avatar_url": "<avatar_url>"
             }
         ],
-        "next_token": "100"
+        "next_token": "100",
+        "total": 200
     }
 
+To paginate, check for ``next_token`` and if present, call the endpoint again
+with ``from`` set to the value of ``next_token``. This will return a new page.
+
+If the endpoint does not return a ``next_token`` then there are no more users
+to paginate through.
 
 Query Account
 =============

docs/application_services.md

@@ -23,9 +23,13 @@ namespaces:
   users:  # List of users we're interested in
     - exclusive: <bool>
       regex: <regex>
+      group_id: <group>
     - ...
   aliases: []  # List of aliases we're interested in
   rooms: [] # List of room ids we're interested in
 ```
 
+`exclusive`: If enabled, only this application service is allowed to register users in its namespace(s).
+`group_id`: All users of this application service are dynamically joined to this group. This is useful for e.g user organisation or flairs.
+
 See the [spec](https://matrix.org/docs/spec/application_service/unstable.html) for further details on how application services work.

docs/metrics-howto.md

@@ -60,6 +60,31 @@
 
 1.  Restart Prometheus.
 
+## Monitoring workers
+
+To monitor a Synapse installation using
+[workers](https://github.com/matrix-org/synapse/blob/master/docs/workers.md),
+every worker needs to be monitored independently, in addition to
+the main homeserver process. This is because workers don't send
+their metrics to the main homeserver process, but expose them
+directly (if they are configured to do so).
+
+To allow collecting metrics from a worker, you need to add a
+`metrics` listener to its configuration, by adding the following
+under `worker_listeners`:
+
+```yaml
+ - type: metrics
+   bind_address: ''
+   port: 9101
+```
+
+The `bind_address` and `port` parameters should be set so that
+the resulting listener can be reached by prometheus, and they
+don't clash with an existing worker.
+With this example, the worker's metrics would then be available
+on `http://127.0.0.1:9101`.
+
 ## Renaming of metrics & deprecation of old names in 1.2
 
 Synapse 1.2 updates the Prometheus metrics to match the naming

docs/sample_config.yaml

@@ -1518,6 +1518,30 @@ sso:
     #
     #     * server_name: the homeserver's name.
     #
+    # * HTML page which notifies the user that they are authenticating to confirm
+    #   an operation on their account during the user interactive authentication
+    #   process: 'sso_auth_confirm.html'.
+    #
+    #   When rendering, this template is given the following variables:
+    #     * redirect_url: the URL the user is about to be redirected to. Needs
+    #                     manual escaping (see
+    #                     https://jinja.palletsprojects.com/en/2.11.x/templates/#html-escaping).
+    #
+    #     * description: the operation which the user is being asked to confirm
+    #
+    # * HTML page shown after a successful user interactive authentication session:
+    #   'sso_auth_success.html'.
+    #
+    #   Note that this page must include the JavaScript which notifies of a successful authentication
+    #   (see https://matrix.org/docs/spec/client_server/r0.6.0#fallback).
+    #
+    #   This template has no additional variables.
+    #
+    # * HTML page shown during single sign-on if a deactivated user (according to Synapse's database)
+    #   attempts to login: 'sso_account_deactivated.html'.
+    #
+    #   This template has no additional variables.
+    #
     # You can see the default templates at:
     # https://github.com/matrix-org/synapse/tree/master/synapse/res/templates
     #