Merge remote-tracking branch 'origin/develop' into clokep/cache-types

matrix-org · Feb 22, 2021 · 2784cda · 2784cda
2 parents 7cce385 + e22b718
commit 2784cda
Show file tree

Hide file tree

Showing 82 changed files with 626 additions and 153 deletions.
diff --git a/CHANGES.md b/CHANGES.md
@@ -1,5 +1,5 @@
-Synapse 1.xx.0 (2021-xx-xx)
-===========================
+Synapse 1.28.0rc1 (2021-02-19)
+==============================
 
 Note that this release drops support for ARMv7 in the official Docker images, due to repeated problems building for ARMv7 (and the associated maintenance burden this entails).
 
@@ -10,13 +10,83 @@ Removal warning
 
 The v1 list accounts API is deprecated and will be removed in a future release.
 This API was undocumented and misleading. It can be replaced by the
-[v2 list accounts API](https://github.com/matrix-org/synapse/blob/master/docs/admin_api/user_admin_api.rst#list-accounts),
+[v2 list accounts API](https://github.com/matrix-org/synapse/blob/release-v1.28.0/docs/admin_api/user_admin_api.rst#list-accounts),
 which has been available since Synapse 1.7.0 (2019-12-13).
 
 Please check if you're using any scripts which use the admin API and replace
 `GET /_synapse/admin/v1/users/<user_id>` with `GET /_synapse/admin/v2/users`.
 
 
+Features
+--------
+
+- New admin API to get the context of an event: `/_synapse/admin/rooms/{roomId}/context/{eventId}`. ([\#9150](https://github.com/matrix-org/synapse/issues/9150))
+- Further improvements to the user experience of registration via single sign-on. ([\#9300](https://github.com/matrix-org/synapse/issues/9300), [\#9301](https://github.com/matrix-org/synapse/issues/9301))
+- Add hook to spam checker modules that allow checking file uploads and remote downloads. ([\#9311](https://github.com/matrix-org/synapse/issues/9311))
+- Add support for receiving OpenID Connect authentication responses via form `POST`s rather than `GET`s. ([\#9376](https://github.com/matrix-org/synapse/issues/9376))
+- Add the shadow-banning status to the admin API for user info. ([\#9400](https://github.com/matrix-org/synapse/issues/9400))
+
+
+Bugfixes
+--------
+
+- Fix long-standing bug where sending email notifications would fail for rooms that the server had since left. ([\#9257](https://github.com/matrix-org/synapse/issues/9257))
+- Fix bug in Synapse 1.27.0rc1 which meant the "session expired" error page during SSO registration was badly formatted. ([\#9296](https://github.com/matrix-org/synapse/issues/9296))
+- Assert a maximum length for some parameters for spec compliance. ([\#9321](https://github.com/matrix-org/synapse/issues/9321), [\#9393](https://github.com/matrix-org/synapse/issues/9393))
+- Fix additional errors when previewing URLs: "AttributeError 'NoneType' object has no attribute 'xpath'" and "ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.". ([\#9333](https://github.com/matrix-org/synapse/issues/9333))
+- Fix a bug causing Synapse to impose the wrong type constraints on fields when processing responses from appservices to `/_matrix/app/v1/thirdparty/user/{protocol}`. ([\#9361](https://github.com/matrix-org/synapse/issues/9361))
+- Fix bug where Synapse would occasionally stop reconnecting to Redis after the connection was lost. ([\#9391](https://github.com/matrix-org/synapse/issues/9391))
+- Fix a long-standing bug when upgrading a room: "TypeError: '>' not supported between instances of 'NoneType' and 'int'". ([\#9395](https://github.com/matrix-org/synapse/issues/9395))
+- Reduce the amount of memory used when generating the URL preview of a file that is larger than the `max_spider_size`. ([\#9421](https://github.com/matrix-org/synapse/issues/9421))
+- Fix a long-standing bug in the deduplication of old presence, resulting in no deduplication. ([\#9425](https://github.com/matrix-org/synapse/issues/9425))
+- The `ui_auth.session_timeout` config option can now be specified in terms of number of seconds/minutes/etc/. Contributed by Rishabh Arya. ([\#9426](https://github.com/matrix-org/synapse/issues/9426))
+- Fix a bug introduced in v1.27.0: "TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType." related to the user directory. ([\#9428](https://github.com/matrix-org/synapse/issues/9428))
+
+
+Updates to the Docker image
+---------------------------
+
+- Drop support for ARMv7 in Docker images. ([\#9433](https://github.com/matrix-org/synapse/issues/9433))
+
+
+Improved Documentation
+----------------------
+
+- Reorganize CHANGELOG.md. ([\#9281](https://github.com/matrix-org/synapse/issues/9281))
+- Add note to `auto_join_rooms` config option explaining existing rooms must be publicly joinable. ([\#9291](https://github.com/matrix-org/synapse/issues/9291))
+- Correct name of Synapse's service file in TURN howto. ([\#9308](https://github.com/matrix-org/synapse/issues/9308))
+- Fix the braces in the `oidc_providers` section of the sample config. ([\#9317](https://github.com/matrix-org/synapse/issues/9317))
+- Update installation instructions on Fedora. ([\#9322](https://github.com/matrix-org/synapse/issues/9322))
+- Add HTTP/2 support to the nginx example configuration. Contributed by David Vo. ([\#9390](https://github.com/matrix-org/synapse/issues/9390))
+- Update docs for using Gitea as OpenID provider. ([\#9404](https://github.com/matrix-org/synapse/issues/9404))
+- Document that pusher instances are shardable. ([\#9407](https://github.com/matrix-org/synapse/issues/9407))
+- Fix erroneous documentation from v1.27.0 about updating the SAML2 callback URL. ([\#9434](https://github.com/matrix-org/synapse/issues/9434))
+
+
+Deprecations and Removals
+-------------------------
+
+- Deprecate old admin API `GET /_synapse/admin/v1/users/<user_id>`. ([\#9429](https://github.com/matrix-org/synapse/issues/9429))
+
+
+Internal Changes
+----------------
+
+- Fix 'object name reserved for internal use' errors with recent versions of SQLite. ([\#9003](https://github.com/matrix-org/synapse/issues/9003))
+- Add experimental support for running Synapse with PyPy. ([\#9123](https://github.com/matrix-org/synapse/issues/9123))
+- Deny access to additional IP addresses by default. ([\#9240](https://github.com/matrix-org/synapse/issues/9240))
+- Update the `Cursor` type hints to better match PEP 249. ([\#9299](https://github.com/matrix-org/synapse/issues/9299))
+- Add debug logging for SRV lookups. Contributed by @Bubu. ([\#9305](https://github.com/matrix-org/synapse/issues/9305))
+- Improve logging for OIDC login flow. ([\#9307](https://github.com/matrix-org/synapse/issues/9307))
+- Share the code for handling required attributes between the CAS and SAML handlers. ([\#9326](https://github.com/matrix-org/synapse/issues/9326))
+- Clean up the code to load the metadata for OpenID Connect identity providers. ([\#9362](https://github.com/matrix-org/synapse/issues/9362))
+- Convert tests to use `HomeserverTestCase`. ([\#9377](https://github.com/matrix-org/synapse/issues/9377), [\#9396](https://github.com/matrix-org/synapse/issues/9396))
+- Update the version of black used to 20.8b1. ([\#9381](https://github.com/matrix-org/synapse/issues/9381))
+- Allow OIDC config to override discovered values. ([\#9384](https://github.com/matrix-org/synapse/issues/9384))
+- Remove some dead code from the acceptance of room invites path. ([\#9394](https://github.com/matrix-org/synapse/issues/9394))
+- Clean up an unused method in the presence handler code. ([\#9408](https://github.com/matrix-org/synapse/issues/9408))
+
+
 Synapse 1.27.0 (2021-02-16)
 ===========================
 

diff --git a/changelog.d/8957.feature b/changelog.d/8957.feature
@@ -0,0 +1 @@
+Add rate limiters to cross-user key sharing requests.
diff --git a/changelog.d/9003.misc b/changelog.d/9003.misc
diff --git a/changelog.d/9123.misc b/changelog.d/9123.misc
diff --git a/changelog.d/9150.feature b/changelog.d/9150.feature
diff --git a/changelog.d/9203.feature b/changelog.d/9203.feature
@@ -0,0 +1 @@
+Add some configuration settings to make users' profile data more private.
diff --git a/changelog.d/9240.misc b/changelog.d/9240.misc
diff --git a/changelog.d/9257.bugfix b/changelog.d/9257.bugfix
diff --git a/changelog.d/9281.doc b/changelog.d/9281.doc
diff --git a/changelog.d/9291.doc b/changelog.d/9291.doc
diff --git a/changelog.d/9296.bugfix b/changelog.d/9296.bugfix
diff --git a/changelog.d/9299.misc b/changelog.d/9299.misc
diff --git a/changelog.d/9300.feature b/changelog.d/9300.feature
diff --git a/changelog.d/9301.feature b/changelog.d/9301.feature
diff --git a/changelog.d/9305.misc b/changelog.d/9305.misc
diff --git a/changelog.d/9307.misc b/changelog.d/9307.misc
diff --git a/changelog.d/9308.doc b/changelog.d/9308.doc
diff --git a/changelog.d/9311.feature b/changelog.d/9311.feature
diff --git a/changelog.d/9317.doc b/changelog.d/9317.doc
diff --git a/changelog.d/9321.bugfix b/changelog.d/9321.bugfix
diff --git a/changelog.d/9322.doc b/changelog.d/9322.doc
diff --git a/changelog.d/9326.misc b/changelog.d/9326.misc
diff --git a/changelog.d/9333.bugfix b/changelog.d/9333.bugfix
diff --git a/changelog.d/9361.bugfix b/changelog.d/9361.bugfix
diff --git a/changelog.d/9362.misc b/changelog.d/9362.misc
diff --git a/changelog.d/9376.feature b/changelog.d/9376.feature
diff --git a/changelog.d/9377.misc b/changelog.d/9377.misc
diff --git a/changelog.d/9381.misc b/changelog.d/9381.misc
diff --git a/changelog.d/9383.feature b/changelog.d/9383.feature
@@ -0,0 +1 @@
+Add a configuration option, `user_directory.prefer_local_users`, which when enabled will make it more likely for users on the same server as you to appear above other users.
diff --git a/changelog.d/9384.misc b/changelog.d/9384.misc
diff --git a/changelog.d/9385.feature b/changelog.d/9385.feature
@@ -0,0 +1 @@
+ Add a configuration option, `user_directory.prefer_local_users`, which when enabled will make it more likely for users on the same server as you to appear above other users.
diff --git a/changelog.d/9390.doc b/changelog.d/9390.doc
diff --git a/changelog.d/9391.bugfix b/changelog.d/9391.bugfix
diff --git a/changelog.d/9393.bugfix b/changelog.d/9393.bugfix
diff --git a/changelog.d/9394.misc b/changelog.d/9394.misc
diff --git a/changelog.d/9395.bugfix b/changelog.d/9395.bugfix
diff --git a/changelog.d/9396.misc b/changelog.d/9396.misc
diff --git a/changelog.d/9400.feature b/changelog.d/9400.feature
diff --git a/changelog.d/9402.bugfix b/changelog.d/9402.bugfix
@@ -0,0 +1 @@
+Fix a bug where a lot of unnecessary presence updates were sent when joining a room.
diff --git a/changelog.d/9404.doc b/changelog.d/9404.doc
diff --git a/changelog.d/9407.doc b/changelog.d/9407.doc
diff --git a/changelog.d/9408.misc b/changelog.d/9408.misc
diff --git a/changelog.d/9421.bugfix b/changelog.d/9421.bugfix
diff --git a/changelog.d/9425.bugfix b/changelog.d/9425.bugfix
diff --git a/changelog.d/9426.bugfix b/changelog.d/9426.bugfix
diff --git a/changelog.d/9428.bugfix b/changelog.d/9428.bugfix
diff --git a/changelog.d/9429.removal b/changelog.d/9429.removal
diff --git a/changelog.d/9432.misc b/changelog.d/9432.misc
@@ -0,0 +1 @@
+Add documentation and type hints to `parse_duration`.
diff --git a/changelog.d/9433.docker b/changelog.d/9433.docker
diff --git a/changelog.d/9434.doc b/changelog.d/9434.doc
diff --git a/changelog.d/9438.feature b/changelog.d/9438.feature
@@ -0,0 +1 @@
+Add support for regenerating thumbnails if they have been deleted but the original image is still stored.
diff --git a/changelog.d/9440.bugfix b/changelog.d/9440.bugfix
@@ -0,0 +1 @@
+Fix bug introduced in v1.27.0 where allowing a user to choose their own username when logging in via single sign-on did not work unless an `idp_icon` was defined.
diff --git a/docs/sample_config.yaml b/docs/sample_config.yaml
@@ -101,6 +101,14 @@ pid_file: DATADIR/homeserver.pid
 #
 #limit_profile_requests_to_users_who_share_rooms: true
 
+# Uncomment to prevent a user's profile data from being retrieved and
+# displayed in a room until they have joined it. By default, a user's
+# profile data is included in an invite event, regardless of the values
+# of the above two settings, and whether or not the users share a server.
+# Defaults to 'true'.
+#
+#include_profile_data_on_invite: false
+
 # If set to 'true', removes the need for authentication to access the server's
 # public rooms directory through the client API, meaning that anyone can
 # query the room directory. Defaults to 'false'.
@@ -699,6 +707,12 @@ acme:
 #  - matrix.org
 #  - example.com
 
+# Uncomment to disable profile lookup over federation. By default, the
+# Federation API allows other homeservers to obtain profile data of any user
+# on this homeserver. Defaults to 'true'.
+#
+#allow_profile_lookup_over_federation: false
+
 
 ## Caching ##
 
@@ -2530,19 +2544,35 @@ spam_checker:
 
 # User Directory configuration
 #
-# 'enabled' defines whether users can search the user directory. If
-# false then empty responses are returned to all queries. Defaults to
-# true.
-#
-# 'search_all_users' defines whether to search all users visible to your HS
-# when searching the user directory, rather than limiting to users visible
-# in public rooms.  Defaults to false.  If you set it True, you'll have to
-# rebuild the user_directory search indexes, see
-# https://github.com/matrix-org/synapse/blob/master/docs/user_directory.md
-#
-#user_directory:
-#  enabled: true
-#  search_all_users: false
+user_directory:
+    # Defines whether users can search the user directory. If false then
+    # empty responses are returned to all queries. Defaults to true.
+    #
+    # Uncomment to disable the user directory.
+    #
+    #enabled: false
+
+    # Defines whether to search all users visible to your HS when searching
+    # the user directory, rather than limiting to users visible in public
+    # rooms. Defaults to false.
+    #
+    # If you set it true, you'll have to rebuild the user_directory search
+    # indexes, see:
+    # https://github.com/matrix-org/synapse/blob/master/docs/user_directory.md
+    #
+    # Uncomment to return search results containing all known users, even if that
+    # user does not share a room with the requester.
+    #
+    #search_all_users: true
+
+    # Defines whether to prefer local users in search query results.
+    # If True, local users are more likely to appear above remote users
+    # when searching the user directory. Defaults to false.
+    #
+    # Uncomment to prefer local over remote users in user directory search
+    # results.
+    #
+    #prefer_local_users: true
 
 
 # User Consent configuration

diff --git a/synapse/__init__.py b/synapse/__init__.py
@@ -48,7 +48,7 @@
 except ImportError:
     pass
 
-__version__ = "1.27.0"
+__version__ = "1.28.0rc1"
 
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
     # We import here so that we don't have to install a bunch of deps when

diff --git a/synapse/api/constants.py b/synapse/api/constants.py
@@ -98,11 +98,14 @@ class EventTypes:
 
     Retention = "m.room.retention"
 
-    Presence = "m.presence"
-
     Dummy = "org.matrix.dummy_event"
 
 
+class EduTypes:
+    Presence = "m.presence"
+    RoomKeyRequest = "m.room_key_request"
+
+
 class RejectedReason:
     AUTH_ERROR = "auth_error"
 

diff --git a/synapse/api/ratelimiting.py b/synapse/api/ratelimiting.py
@@ -14,7 +14,7 @@
 # limitations under the License.
 
 from collections import OrderedDict
-from typing import Any, Optional, Tuple
+from typing import Hashable, Optional, Tuple
 
 from synapse.api.errors import LimitExceededError
 from synapse.types import Requester
@@ -42,7 +42,9 @@ def __init__(self, clock: Clock, rate_hz: float, burst_count: int):
         #   * How many times an action has occurred since a point in time
         #   * The point in time
         #   * The rate_hz of this particular entry. This can vary per request
-        self.actions = OrderedDict()  # type: OrderedDict[Any, Tuple[float, int, float]]
+        self.actions = (
+            OrderedDict()
+        )  # type: OrderedDict[Hashable, Tuple[float, int, float]]
 
     def can_requester_do_action(
         self,
@@ -82,7 +84,7 @@ def can_requester_do_action(
 
     def can_do_action(
         self,
-        key: Any,
+        key: Hashable,
         rate_hz: Optional[float] = None,
         burst_count: Optional[int] = None,
         update: bool = True,
@@ -175,7 +177,7 @@ def _prune_message_counts(self, time_now_s: int):
 
     def ratelimit(
         self,
-        key: Any,
+        key: Hashable,
         rate_hz: Optional[float] = None,
         burst_count: Optional[int] = None,
         update: bool = True,

diff --git a/synapse/config/_base.py b/synapse/config/_base.py
@@ -21,7 +21,7 @@
 from collections import OrderedDict
 from hashlib import sha256
 from textwrap import dedent
-from typing import Any, Iterable, List, MutableMapping, Optional
+from typing import Any, Iterable, List, MutableMapping, Optional, Union
 
 import attr
 import jinja2
@@ -147,7 +147,20 @@ def parse_size(value):
         return int(value) * size
 
     @staticmethod
-    def parse_duration(value):
+    def parse_duration(value: Union[str, int]) -> int:
+        """Convert a duration as a string or integer to a number of milliseconds.
+
+        If an integer is provided it is treated as milliseconds and is unchanged.
+
+        String durations can have a suffix of 's', 'm', 'h', 'd', 'w', or 'y'.
+        No suffix is treated as milliseconds.
+
+        Args:
+            value: The duration to parse.
+
+        Returns:
+            The number of milliseconds in the duration.
+        """
         if isinstance(value, int):
             return value
         second = 1000

diff --git a/synapse/config/federation.py b/synapse/config/federation.py
@@ -41,6 +41,10 @@ def read_config(self, config, **kwargs):
         )
         self.federation_metrics_domains = set(federation_metrics_domains)
 
+        self.allow_profile_lookup_over_federation = config.get(
+            "allow_profile_lookup_over_federation", True
+        )
+
     def generate_config_section(self, config_dir_path, server_name, **kwargs):
         return """\
         ## Federation ##
@@ -66,6 +70,12 @@ def generate_config_section(self, config_dir_path, server_name, **kwargs):
         #federation_metrics_domains:
         #  - matrix.org
         #  - example.com
+
+        # Uncomment to disable profile lookup over federation. By default, the
+        # Federation API allows other homeservers to obtain profile data of any user
+        # on this homeserver. Defaults to 'true'.
+        #
+        #allow_profile_lookup_over_federation: false
         """