Improve get auth chain difference algorithm. #7095

erikjohnston · 2020-03-17T16:20:18Z

It was originally implemented by pulling the full auth chain of all
state sets out of the database and doing set comparison. However, that
can take a lot work if the state and auth chains are large.

Instead, lets try and fetch the auth chains at the same time and
calcualte the difference on the fly, allowing us to bail early if all
the auth chains converge. Assuming that the auth chains do converge more
often than not, this should improve performance. Hopefully.

Relates to: #6943

It was originally implemented by pulling the full auth chain of all state sets out of the database and doing set comparison. However, that can take a lot work if the state and auth chains are large. Instead, lets try and fetch the auth chains at the same time and calcualte the difference on the fly, allowing us to bail early if all the auth chains converge. Assuming that the auth chains do converge more often than not, this should improve performance. Hopefully.

synapse/storage/data_stores/main/event_federation.py

ara4n · 2020-03-17T18:58:46Z

synapse/storage/data_stores/main/event_federation.py

+            for event_id in initial_events
+        }
+
+        # We need to get the depth of the initial events for sorting purposes.


can we trust this value of depth?

Nope! Though hopefully as explained above we're using it as a hint, and so worst case if its wrong we basically pull the full auth chains out of the DB as we're currently doing

tests/storage/test_event_federation.py

Co-Authored-By: Matthew Hodgson <matthew@matrix.org>

richvdh

well, it looks pretty plausible to me. A few suggestions for clarity and efficiency.

synapse/state/__init__.py

richvdh · 2020-03-18T11:22:11Z

synapse/state/v2.py

-                    (EventTypes.PowerLevels, ""),
-                    (EventTypes.Create, ""),
-                    (EventTypes.JoinRules, ""),
+        auth_sets.append(


these aren't really auth sets as I understand the term? consider a rename?

Also, a comment to explain why we're looking at these event types specifically.

So this is a colossal hack that basically notices that we re-add all the conflicted state later, and only care about events that are "auth" events (i.e. types that are required to auth other events). Given its a bit of a mind bend to understand that this is OK, and the vast majority of state is memberships, I think its probably better to just remove this filtering entirely.

synapse/storage/data_stores/main/event_federation.py

richvdh · 2020-03-18T12:06:29Z

synapse/storage/data_stores/main/event_federation.py

+
+            # Fetch the auth events and their depths of the N last events we're
+            # currently walking
+            search, chunk = search[:-100], search[-100:]


vaguely wondering if a heap would be more efficient for search. let's not rewrite it now though.

Me too, but heapq seems to only let you pop items one at a time I think? Which sounds inefficient

agreed. we might have to roll our own impl, which may or may not work out as a winner over a sorted list.

synapse/storage/data_stores/main/event_federation.py

richvdh

lgtm otherwise

synapse/storage/data_stores/main/event_federation.py

Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

@devec0

Synapse 1.12.0 (2020-03-23) =========================== No significant changes since 1.12.0rc1. Debian packages and Docker images are rebuilt using the latest versions of dependency libraries, including Twisted 20.3.0. **Please see security advisory below**. Security advisory ----------------- Synapse may be vulnerable to request-smuggling attacks when it is used with a reverse-proxy. The vulnerabilties are fixed in Twisted 20.3.0, and are described in [CVE-2020-10108](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10108) and [CVE-2020-10109](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10109). For a good introduction to this class of request-smuggling attacks, see https://portswigger.net/research/http-desync-attacks-request-smuggling-reborn. We are not aware of these vulnerabilities being exploited in the wild, and do not believe that they are exploitable with current versions of any reverse proxies. Nevertheless, we recommend that all Synapse administrators ensure that they have the latest versions of the Twisted library to ensure that their installation remains secure. * Administrators using the [`matrix.org` Docker image](https://hub.docker.com/r/matrixdotorg/synapse/) or the [Debian/Ubuntu packages from `matrix.org`](https://github.com/matrix-org/synapse/blob/master/INSTALL.md#matrixorg-packages) should ensure that they have version 1.12.0 installed: these images include Twisted 20.3.0. * Administrators who have [installed Synapse from source](https://github.com/matrix-org/synapse/blob/master/INSTALL.md#installing-from-source) should upgrade Twisted within their virtualenv by running: ```sh <path_to_virtualenv>/bin/pip install 'Twisted>=20.3.0' ``` * Administrators who have installed Synapse from distribution packages should consult the information from their distributions. The `matrix.org` Synapse instance was not vulnerable to these vulnerabilities. Advance notice of change to the default `git` branch for Synapse ---------------------------------------------------------------- Currently, the default `git` branch for Synapse is `master`, which tracks the latest release. After the release of Synapse 1.13.0, we intend to change this default to `develop`, which is the development tip. This is more consistent with common practice and modern `git` usage. Although we try to keep `develop` in a stable state, there may be occasions where regressions creep in. Developers and distributors who have scripts which run builds using the default branch of `Synapse` should therefore consider pinning their scripts to `master`. Synapse 1.12.0rc1 (2020-03-19) ============================== Features -------- - Changes related to room alias management ([MSC2432](matrix-org/matrix-spec-proposals#2432)): - Publishing/removing a room from the room directory now requires the user to have a power level capable of modifying the canonical alias, instead of the room aliases. ([\#6965](#6965)) - Validate the `alt_aliases` property of canonical alias events. ([\#6971](#6971)) - Users with a power level sufficient to modify the canonical alias of a room can now delete room aliases. ([\#6986](#6986)) - Implement updated authorization rules and redaction rules for aliases events, from [MSC2261](matrix-org/matrix-spec-proposals#2261) and [MSC2432](matrix-org/matrix-spec-proposals#2432). ([\#7037](#7037)) - Stop sending m.room.aliases events during room creation and upgrade. ([\#6941](#6941)) - Synapse no longer uses room alias events to calculate room names for push notifications. ([\#6966](#6966)) - The room list endpoint no longer returns a list of aliases. ([\#6970](#6970)) - Remove special handling of aliases events from [MSC2260](matrix-org/matrix-spec-proposals#2260) added in v1.10.0rc1. ([\#7034](#7034)) - Expose the `synctl`, `hash_password` and `generate_config` commands in the snapcraft package. Contributed by @devec0. ([\#6315](#6315)) - Check that server_name is correctly set before running database updates. ([\#6982](#6982)) - Break down monthly active users by `appservice_id` and emit via Prometheus. ([\#7030](#7030)) - Render a configurable and comprehensible error page if something goes wrong during the SAML2 authentication process. ([\#7058](#7058), [\#7067](#7067)) - Add an optional parameter to control whether other sessions are logged out when a user's password is modified. ([\#7085](#7085)) - Add prometheus metrics for the number of active pushers. ([\#7103](#7103), [\#7106](#7106)) - Improve performance when making HTTPS requests to sygnal, sydent, etc, by sharing the SSL context object between connections. ([\#7094](#7094)) Bugfixes -------- - When a user's profile is updated via the admin API, also generate a displayname/avatar update for that user in each room. ([\#6572](#6572)) - Fix a couple of bugs in email configuration handling. ([\#6962](#6962)) - Fix an issue affecting worker-based deployments where replication would stop working, necessitating a full restart, after joining a large room. ([\#6967](#6967)) - Fix `duplicate key` error which was logged when rejoining a room over federation. ([\#6968](#6968)) - Prevent user from setting 'deactivated' to anything other than a bool on the v2 PUT /users Admin API. ([\#6990](#6990)) - Fix py35-old CI by using native tox package. ([\#7018](#7018)) - Fix a bug causing `org.matrix.dummy_event` to be included in responses from `/sync`. ([\#7035](#7035)) - Fix a bug that renders UTF-8 text files incorrectly when loaded from media. Contributed by @TheStranjer. ([\#7044](#7044)) - Fix a bug that would cause Synapse to respond with an error about event visibility if a client tried to request the state of a room at a given token. ([\#7066](#7066)) - Repair a data-corruption issue which was introduced in Synapse 1.10, and fixed in Synapse 1.11, and which could cause `/sync` to return with 404 errors about missing events and unknown rooms. ([\#7070](#7070)) - Fix a bug causing account validity renewal emails to be sent even if the feature is turned off in some cases. ([\#7074](#7074)) Improved Documentation ---------------------- - Updated CentOS8 install instructions. Contributed by Richard Kellner. ([\#6925](#6925)) - Fix `POSTGRES_INITDB_ARGS` in the `contrib/docker/docker-compose.yml` example docker-compose configuration. ([\#6984](#6984)) - Change date in [INSTALL.md](./INSTALL.md#tls-certificates) for last date of getting TLS certificates to November 2019. ([\#7015](#7015)) - Document that the fallback auth endpoints must be routed to the same worker node as the register endpoints. ([\#7048](#7048)) Deprecations and Removals ------------------------- - Remove the unused query_auth federation endpoint per [MSC2451](matrix-org/matrix-spec-proposals#2451). ([\#7026](#7026)) Internal Changes ---------------- - Add type hints to `logging/context.py`. ([\#6309](#6309)) - Add some clarifications to `README.md` in the database schema directory. ([\#6615](#6615)) - Refactoring work in preparation for changing the event redaction algorithm. ([\#6874](#6874), [\#6875](#6875), [\#6983](#6983), [\#7003](#7003)) - Improve performance of v2 state resolution for large rooms. ([\#6952](#6952), [\#7095](#7095)) - Reduce time spent doing GC, by freezing objects on startup. ([\#6953](#6953)) - Minor perfermance fixes to `get_auth_chain_ids`. ([\#6954](#6954)) - Don't record remote cross-signing keys in the `devices` table. ([\#6956](#6956)) - Use flake8-comprehensions to enforce good hygiene of list/set/dict comprehensions. ([\#6957](#6957)) - Merge worker apps together. ([\#6964](#6964), [\#7002](#7002), [\#7055](#7055), [\#7104](#7104)) - Remove redundant `store_room` call from `FederationHandler._process_received_pdu`. ([\#6979](#6979)) - Update warning for incorrect database collation/ctype to include link to documentation. ([\#6985](#6985)) - Add some type annotations to the database storage classes. ([\#6987](#6987)) - Port `synapse.handlers.presence` to async/await. ([\#6991](#6991), [\#7019](#7019)) - Add some type annotations to the federation base & client classes. ([\#6995](#6995)) - Port `synapse.rest.keys` to async/await. ([\#7020](#7020)) - Add a type check to `is_verified` when processing room keys. ([\#7045](#7045)) - Add type annotations and comments to the auth handler. ([\#7063](#7063))

It was originally implemented by pulling the full auth chain of all state sets out of the database and doing set comparison. However, that can take a lot work if the state and auth chains are large. Instead, lets try and fetch the auth chains at the same time and calculate the difference on the fly, allowing us to bail early if all the auth chains converge. Assuming that the auth chains do converge more often than not, this should improve performance. Hopefully.

* commit '4a17a647a': Improve get auth chain difference algorithm. (#7095)

erikjohnston added 4 commits March 17, 2020 13:08

Port event federation tests to new style

d128efc

Newsfile

91a6f78

Fix unit tests

0ebf292

erikjohnston requested a review from a team March 17, 2020 16:40

Really fix unit tests

bd2e18a

ara4n reviewed Mar 17, 2020

View reviewed changes

synapse/storage/data_stores/main/event_federation.py Outdated Show resolved Hide resolved

ara4n reviewed Mar 17, 2020

View reviewed changes

synapse/storage/data_stores/main/event_federation.py Outdated Show resolved Hide resolved

ara4n reviewed Mar 17, 2020

View reviewed changes

tests/storage/test_event_federation.py Outdated Show resolved Hide resolved

Fix typos

d373279

Co-Authored-By: Matthew Hodgson <matthew@matrix.org>

richvdh reviewed Mar 18, 2020

View reviewed changes

Review comments

fa04467

erikjohnston requested a review from richvdh March 18, 2020 14:08

richvdh approved these changes Mar 18, 2020

View reviewed changes

synapse/storage/data_stores/main/event_federation.py Outdated Show resolved Hide resolved

synapse/storage/data_stores/main/event_federation.py Outdated Show resolved Hide resolved

erikjohnston and others added 2 commits March 18, 2020 15:14

Apply suggestions from code review

f47a417

Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

pep8

b209466

erikjohnston merged commit 4a17a64 into develop Mar 18, 2020

erikjohnston deleted the erikj/faster_v2_state branch March 18, 2020 16:46

erikjohnston mentioned this pull request Mar 19, 2020

State resolution on big rooms can get very CPU-hungry #6943

Closed

richvdh mentioned this pull request Apr 14, 2020

Fix a potentially-huge sql query #7274

Merged

babolivier pushed a commit that referenced this pull request Sep 1, 2021

Improve get auth chain difference algorithm. (#7095)

62c75e6

* commit '4a17a647a': Improve get auth chain difference algorithm. (#7095)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve get auth chain difference algorithm. #7095

Improve get auth chain difference algorithm. #7095

erikjohnston commented Mar 17, 2020

ara4n Mar 17, 2020

erikjohnston Mar 18, 2020

richvdh left a comment

richvdh Mar 18, 2020

erikjohnston Mar 18, 2020

richvdh Mar 18, 2020

erikjohnston Mar 18, 2020

richvdh Mar 18, 2020

richvdh left a comment

Improve get auth chain difference algorithm. #7095

Improve get auth chain difference algorithm. #7095

Conversation

erikjohnston commented Mar 17, 2020

ara4n Mar 17, 2020

Choose a reason for hiding this comment

erikjohnston Mar 18, 2020

Choose a reason for hiding this comment

richvdh left a comment

Choose a reason for hiding this comment

richvdh Mar 18, 2020

Choose a reason for hiding this comment

erikjohnston Mar 18, 2020

Choose a reason for hiding this comment

richvdh Mar 18, 2020

Choose a reason for hiding this comment

erikjohnston Mar 18, 2020

Choose a reason for hiding this comment

richvdh Mar 18, 2020

Choose a reason for hiding this comment

richvdh left a comment

Choose a reason for hiding this comment