Implement push rule evaluation in Rust. #13838

erikjohnston · 2022-09-19T11:20:00Z

Builds on top of #13768

Co-authored-by: David Robertson <davidr@element.io>

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

erikjohnston · 2022-09-19T11:22:14Z

rust/src/push/evaluator.rs

+        let known_condition = match condition {
+            Condition::Known(known) => known,
+            Condition::Unknown(_) => {
+                return Ok(false);


Note: we used to return true here for some bizarre reason. I moved it to return "false" as a) that makes way more sense, and b) "invalid" conditions get turned into "Condition::Unknown(_)` and should never match.

I always found this really weird, I wonder if we should land that now in the current code (if we think this is going to take a bit of review cycles before landing) to ensure it doesn't break clients?

To be clear: we were returning true under this condition in the Python code, so this isn't a regression or anything

But now we're returning false? Maybe I'm misreading the code... but I'm mostly going by your comment:

we used to return true here

Which implies we no longer do.

Sorry for not being clear. This is the code currently on develop and in releases:

synapse/synapse/push/push_rule_evaluator.py

Lines 211 to 216 in 87fe9db

else:

# XXX This looks incorrect -- we have reached an unknown condition

# kind and are unconditionally returning that it matches. Note

# that it seems possible to provide a condition to the /pushrules

# endpoint with an unknown kind, see _rule_tuple_from_request_object.

return True

erikjohnston · 2022-09-19T11:36:38Z

rust/src/push/mod.rs

@@ -274,6 +279,8 @@ pub enum KnownCondition {
    #[serde(rename = "org.matrix.msc3772.relation_match")]
    RelationMatch {
        rel_type: Cow<'static, str>,
+        #[serde(skip_serializing_if = "Option::is_none", rename = "type")]
+        event_type_pattern: Option<Cow<'static, str>>,


This should probably actually be in #13768, though I don't think we use it in practice?

erikjohnston · 2022-09-19T11:36:58Z

synapse/push/bulk_push_rule_evaluator.py

+    d: Union[EventBase, Mapping[str, Any]],
+    prefix: Optional[List[str]] = None,
+    result: Optional[Dict[str, str]] = None,
+) -> Dict[str, str]:


This is moved from push_rule_evaluator.py

erikjohnston · 2022-09-19T11:37:01Z

synapse/push/httppusher.py

@@ -56,6 +56,39 @@
 )


+def tweaks_for_actions(actions: List[Union[str, Dict]]) -> Dict[str, Any]:


This is moved from push_rule_evaluator.py

rust/src/push/evaluator.rs

MatMaul · 2022-09-20T07:38:02Z

rust/src/push/evaluator.rs

+        };
+
+        if let Some(value) = self.flattened_keys.get(&*event_match.key) {
+            let compiled_pattern = glob_to_regex(pattern, match_type)?;


Is a cache of compiled patterns would be useful ? Compiling regex tends to be expensive.

Oh, good point. We did have a cache before too. Let me quickly run a benchmark.

Interesting numbers come out of it (annoying I can't commit the benchmark code, as benchmarking doesn't work in a cdylib crate):

test bench_glob_whole ... bench: 28,814 ns/iter (+/- 1,209) test bench_glob_word ... bench: 409,264 ns/iter (+/- 47,114)

So compiling word based regex can take almost half a millisecond on my laptop. If we disable unicode awareness when compiling the regex they end up taking about the same amount of time.

Instead of caching the regex, I've implemented a "fast path", which allows us to avoid compiling the regex unless necessary.

Co-authored-by: Mathieu Velten <mathieuv@matrix.org>

rust/src/push/evaluator.rs

erikjohnston · 2022-09-24T14:48:11Z

A local test of this on postgres takes the time to send a message in a room with 5k local users from ~3s to ~500ms, with the actual calculating of push actions a tiny fraction of that.

erikjohnston · 2022-09-27T14:56:10Z

Note, the files in rust/benches/ are benchmarks, which can be run with cargo bench (though you need nightly).

I'd recommend reviewing this as chunks:

rust/src/push/utils.rs — contains some helper functions that mimic those in Python
rust/src/push/evaluator.rs, stubs/ — the actual evaluator code
The changes to synapse/ — where we use the new evaluator and delete the old one
The benchmarks.

rust/Cargo.toml

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

reivilibre

This looks reasonable to me

rust/src/push/utils.rs

rust/src/push/evaluator.rs

reivilibre · 2022-09-29T11:52:40Z

rust/src/push/evaluator.rs

+    fn match_member_count(&self, is: &str) -> Result<bool, Error> {
+        // The 'is' condition can be things like '>2', '==3' or event just '4'.
+        let captures = INEQUALITY_EXPR.captures(is).context("bad 'is' clause")?;
+        let ineq = captures.get(1).map_or("==", |m| m.as_str());


not sure where the map_or default would actually be used, if ever here?

The regex matches plain numbers, like 3, which are an exact match.

I mean that it looks like the capture group 1 should always exist (even if it's empty string?), so defaulting to == seems superfluous.
e.g. As far as I can tell, captures[1] (maybe with a &-borrow or .as_str() if necessary) does the same thing

Oh, hmm, yes because it will get the empty string

We still need to unwrap the Option so I am somewhat minded to leave it as is?

presumably captures[1] wouldn't be an option, but yes whatever works

rust/src/push/evaluator.rs

synapse/push/bulk_push_rule_evaluator.py

Co-authored-by: reivilibre <oliverw@matrix.org>

reivilibre

LGTM (though not sure if map_or is really necessary, but up to you)

erikjohnston · 2022-09-29T15:12:17Z

Woo, thanks all 🎉

@lukasdenk

Synapse 1.69.0rc1 (2022-10-04) ============================== Please note that legacy Prometheus metric names are now deprecated and will be removed in Synapse 1.73.0. Server administrators should update their dashboards and alerting rules to avoid using the deprecated metric names. See the [upgrade notes](https://matrix-org.github.io/synapse/v1.69/upgrade.html#upgrading-to-v1690) for more details. Features -------- - Allow application services to set the `origin_server_ts` of a state event by providing the query parameter `ts` in [`PUT /_matrix/client/r0/rooms/{roomId}/state/{eventType}/{stateKey}`](https://spec.matrix.org/v1.4/client-server-api/#put_matrixclientv3roomsroomidstateeventtypestatekey), per [MSC3316](matrix-org/matrix-spec-proposals#3316). Contributed by @lukasdenk. ([\#11866](#11866)) - Allow server admins to require a manual approval process before new accounts can be used (using [MSC3866](matrix-org/matrix-spec-proposals#3866)). ([\#13556](#13556)) - Exponentially backoff from backfilling the same event over and over. ([\#13635](#13635), [\#13936](#13936)) - Add cache invalidation across workers to module API. ([\#13667](#13667), [\#13947](#13947)) - Experimental implementation of [MSC3882](matrix-org/matrix-spec-proposals#3882) to allow an existing device/session to generate a login token for use on a new device/session. ([\#13722](#13722), [\#13868](#13868)) - Experimental support for thread-specific receipts ([MSC3771](matrix-org/matrix-spec-proposals#3771)). ([\#13782](#13782), [\#13893](#13893), [\#13932](#13932), [\#13937](#13937), [\#13939](#13939)) - Add experimental support for [MSC3881: Remotely toggle push notifications for another client](matrix-org/matrix-spec-proposals#3881). ([\#13799](#13799), [\#13831](#13831), [\#13860](#13860)) - Keep track when an event pulled over federation fails its signature check so we can intelligently back-off in the future. ([\#13815](#13815)) - Improve validation for the unspecced, internal-only `_matrix/client/unstable/add_threepid/msisdn/submit_token` endpoint. ([\#13832](#13832)) - Faster remote room joins: record _when_ we first partial-join to a room. ([\#13892](#13892)) - Support a `dir` parameter on the `/relations` endpoint per [MSC3715](matrix-org/matrix-spec-proposals#3715). ([\#13920](#13920)) - Ask mail servers receiving emails from Synapse to not send automatic replies (e.g. out-of-office responses). ([\#13957](#13957)) Bugfixes -------- - Send push notifications for invites received over federation. ([\#13719](#13719), [\#14014](#14014)) - Fix a long-standing bug where typing events would be accepted from remote servers not present in a room. Also fix a bug where incoming typing events would cause other incoming events to get stuck during a fast join. ([\#13830](#13830)) - Fix a bug introduced in Synapse v1.53.0 where the experimental implementation of [MSC3715](matrix-org/matrix-spec-proposals#3715) would give incorrect results when paginating forward. ([\#13840](#13840)) - Fix access token leak to logs from proxy agent. ([\#13855](#13855)) - Fix `have_seen_event` cache not being invalidated after we persist an event which causes inefficiency effects like extra `/state` federation calls. ([\#13863](#13863)) - Faster room joins: Fix a bug introduced in 1.66.0 where an error would be logged when syncing after joining a room. ([\#13872](#13872)) - Fix a bug introduced in 1.66.0 where some required fields in the pushrules sent to clients were not present anymore. Contributed by Nico. ([\#13904](#13904)) - Fix packaging to include `Cargo.lock` in `sdist`. ([\#13909](#13909)) - Fix a long-standing bug where device updates could cause delays sending out to-device messages over federation. ([\#13922](#13922)) - Fix a bug introduced in v1.68.0 where Synapse would require `setuptools_rust` at runtime, even though the package is only required at build time. ([\#13952](#13952)) - Fix a long-standing bug where `POST /_matrix/client/v3/keys/query` requests could result in excessively large SQL queries. ([\#13956](#13956)) - Fix a performance regression in the `get_users_in_room` database query. Introduced in v1.67.0. ([\#13972](#13972)) - Fix a bug introduced in v1.68.0 bug where Rust extension wasn't built in `release` mode when using `poetry install`. ([\#14009](#14009)) - Do not return an unspecified `original_event` field when using the stable `/relations` endpoint. Introduced in Synapse v1.57.0. ([\#14025](#14025)) - Correctly handle a race with device lists when a remote user leaves during a partial join. ([\#13885](#13885)) - Correctly handle sending local device list updates to remote servers during a partial join. ([\#13934](#13934)) Improved Documentation ---------------------- - Add `worker_main_http_uri` for the worker generator bash script. ([\#13772](#13772)) - Update URL for the NixOS module for Synapse. ([\#13818](#13818)) - Fix a mistake in sso_mapping_providers.md: `map_user_attributes` is expected to return `display_name`, not `displayname`. ([\#13836](#13836)) - Fix a cross-link from the registration admin API to the `registration_shared_secret` configuration documentation. ([\#13870](#13870)) - Update the man page for the `hash_password` script to correct the default number of bcrypt rounds performed. ([\#13911](#13911), [\#13930](#13930)) - Emphasize the right reasons when to use `(room_id, event_id)` in a database schema. ([\#13915](#13915)) - Add instruction to contributing guide for running unit tests in parallel. Contributed by @ashfame. ([\#13928](#13928)) - Clarify that the `auto_join_rooms` config option can also be used with Space aliases. ([\#13931](#13931)) - Add some cross references to worker documentation. ([\#13974](#13974)) - Linkify urls in config documentation. ([\#14003](#14003)) Deprecations and Removals ------------------------- - Remove the `complete_sso_login` method from the Module API which was deprecated in Synapse 1.13.0. ([\#13843](#13843)) - Announce that legacy metric names are deprecated, will be turned off by default in Synapse v1.71.0 and removed altogether in Synapse v1.73.0. See the upgrade notes for more information. ([\#14024](#14024)) Internal Changes ---------------- - Speed up creation of DM rooms. ([\#13487](#13487), [\#13800](#13800)) - Port push rules to using Rust. ([\#13768](#13768), [\#13838](#13838), [\#13889](#13889)) - Optimise get rooms for user calls. Contributed by Nick @ Beeper (@Fizzadar). ([\#13787](#13787)) - Update the script which makes full schema dumps. ([\#13792](#13792)) - Use shared methods for cache invalidation when persisting events, remove duplicate codepaths. Contributed by Nick @ Beeper (@Fizzadar). ([\#13796](#13796)) - Improve the `synapse.api.auth.Auth` mock used in unit tests. ([\#13809](#13809)) - Faster Remote Room Joins: tell remote homeservers that we are unable to authorise them if they query a room which has partial state on our server. ([\#13823](#13823)) - Carry IdP Session IDs through user-mapping sessions. ([\#13839](#13839)) - Fix the release script not publishing binary wheels. ([\#13850](#13850)) - Raise issue if complement fails with latest deps. ([\#13859](#13859)) - Correct the comments in the complement dockerfile. ([\#13867](#13867)) - Create a new snapshot of the database schema. ([\#13873](#13873)) - Faster room joins: Send device list updates to most servers in rooms with partial state. ([\#13874](#13874), [\#14013](#14013)) - Add comments to the Prometheus recording rules to make it clear which set of rules you need for Grafana or Prometheus Console. ([\#13876](#13876)) - Only pull relevant backfill points from the database based on the current depth and limit (instead of all) every time we want to `/backfill`. ([\#13879](#13879)) - Faster room joins: Avoid waiting for full state when processing `/keys/changes` requests. ([\#13888](#13888)) - Improve backfill robustness by trying more servers when we get a `4xx` error back. ([\#13890](#13890)) - Fix mypy errors with canonicaljson 1.6.3. ([\#13905](#13905)) - Faster remote room joins: correctly handle remote device list updates during a partial join. ([\#13913](#13913)) - Complement image: propagate SIGTERM to all workers. ([\#13914](#13914)) - Update an innaccurate comment in Synapse's upsert database helper. ([\#13924](#13924)) - Update mypy (0.950 -> 0.981) and mypy-zope (0.3.7 -> 0.3.11). ([\#13925](#13925), [\#13993](#13993)) - Use dedicated `get_local_users_in_room(room_id)` function to find local users when calculating users to copy over during a room upgrade. ([\#13960](#13960)) - Refactor language in user directory `_track_user_joined_room` code to make it more clear that we use both local and remote users. ([\#13966](#13966)) - Revert catch-all exceptions being recorded as event pull attempt failures (only handle what we know about). ([\#13969](#13969)) - Speed up calculating push actions in large rooms. ([\#13973](#13973), [\#13992](#13992)) - Enable update notifications from Github's dependabot. ([\#13976](#13976)) - Prototype a workflow to automatically add changelogs to dependabot PRs. ([\#13998](#13998), [\#14011](#14011), [\#14017](#14017), [\#14021](#14021), [\#14027](#14027)) - Fix type annotations to be compatible with new annotations in development versions of twisted. ([\#14012](#14012)) - Clear out stale entries in `event_push_actions_staging` table. ([\#14020](#14020)) - Bump versions of GitHub actions. ([\#13978](#13978), [\#13979](#13979), [\#13980](#13980), [\#13982](#13982), [\#14015](#14015), [\#14019](#14019), [\#14022](#14022), [\#14023](#14023))

@lukasdenk

NOTE: this is absolutely *not* safe for Beeper usage as-is. I have merged all of the Python code in but all our customizations to the base rules and push rule evaluator are not yet present in the new Rust module. This will fail tests as-is and future commits will re-apply our changes in Rust. Synapse 1.69.0 (2022-10-17) =========================== Please note that legacy Prometheus metric names are now deprecated and will be removed in Synapse 1.73.0. Server administrators should update their dashboards and alerting rules to avoid using the deprecated metric names. See the [upgrade notes](https://matrix-org.github.io/synapse/v1.69/upgrade.html#upgrading-to-v1690) for more details. No significant changes since 1.69.0rc4. Synapse 1.69.0rc4 (2022-10-14) ============================== Bugfixes -------- - Fix poor performance of the `event_push_backfill_thread_id` background update, which was introduced in Synapse 1.68.0rc1. ([\matrix-org#14172](matrix-org#14172), [\matrix-org#14181](matrix-org#14181)) Updates to the Docker image --------------------------- - Fix docker build OOMing in CI for arm64 builds. ([\matrix-org#14173](matrix-org#14173)) Synapse 1.69.0rc3 (2022-10-12) ============================== Bugfixes -------- - Fix an issue with Docker images causing the Rust dependencies to not be pinned correctly. Introduced in v1.68.0 ([\matrix-org#14129](matrix-org#14129)) - Fix a bug introduced in Synapse 1.69.0rc1 which would cause registration replication requests to fail if the worker sending the request is not running Synapse 1.69. ([\matrix-org#14135](matrix-org#14135)) - Fix error in background update when rotating existing notifications. Introduced in v1.69.0rc2. ([\matrix-org#14138](matrix-org#14138)) Internal Changes ---------------- - Rename the `url_preview` extra to `url-preview`, for compatability with poetry-core 1.3.0 and [PEP 685](https://peps.python.org/pep-0685/). From-source installations using this extra will need to install using the new name. ([\matrix-org#14085](matrix-org#14085)) Synapse 1.69.0rc2 (2022-10-06) ============================== Deprecations and Removals ------------------------- - Deprecate the `generate_short_term_login_token` method in favor of an async `create_login_token` method in the Module API. ([\matrix-org#13842](matrix-org#13842)) Internal Changes ---------------- - Ensure Synapse v1.69 works with upcoming database changes in v1.70. ([\matrix-org#14045](matrix-org#14045)) - Fix a bug introduced in Synapse v1.68.0 where messages could not be sent in rooms with non-integer `notifications` power level. ([\matrix-org#14073](matrix-org#14073)) - Temporarily pin build-system requirements to workaround an incompatibility with poetry-core 1.3.0. This will be reverted before the v1.69.0 release proper, see [\matrix-org#14079](matrix-org#14079). ([\matrix-org#14080](matrix-org#14080)) Synapse 1.69.0rc1 (2022-10-04) ============================== Features -------- - Allow application services to set the `origin_server_ts` of a state event by providing the query parameter `ts` in [`PUT /_matrix/client/r0/rooms/{roomId}/state/{eventType}/{stateKey}`](https://spec.matrix.org/v1.4/client-server-api/#put_matrixclientv3roomsroomidstateeventtypestatekey), per [MSC3316](matrix-org/matrix-spec-proposals#3316). Contributed by @lukasdenk. ([\matrix-org#11866](matrix-org#11866)) - Allow server admins to require a manual approval process before new accounts can be used (using [MSC3866](matrix-org/matrix-spec-proposals#3866)). ([\matrix-org#13556](matrix-org#13556)) - Exponentially backoff from backfilling the same event over and over. ([\matrix-org#13635](matrix-org#13635), [\matrix-org#13936](matrix-org#13936)) - Add cache invalidation across workers to module API. ([\matrix-org#13667](matrix-org#13667), [\matrix-org#13947](matrix-org#13947)) - Experimental implementation of [MSC3882](matrix-org/matrix-spec-proposals#3882) to allow an existing device/session to generate a login token for use on a new device/session. ([\matrix-org#13722](matrix-org#13722), [\matrix-org#13868](matrix-org#13868)) - Experimental support for thread-specific receipts ([MSC3771](matrix-org/matrix-spec-proposals#3771)). ([\matrix-org#13782](matrix-org#13782), [\matrix-org#13893](matrix-org#13893), [\matrix-org#13932](matrix-org#13932), [\matrix-org#13937](matrix-org#13937), [\matrix-org#13939](matrix-org#13939)) - Add experimental support for [MSC3881: Remotely toggle push notifications for another client](matrix-org/matrix-spec-proposals#3881). ([\matrix-org#13799](matrix-org#13799), [\matrix-org#13831](matrix-org#13831), [\matrix-org#13860](matrix-org#13860)) - Keep track when an event pulled over federation fails its signature check so we can intelligently back-off in the future. ([\matrix-org#13815](matrix-org#13815)) - Improve validation for the unspecced, internal-only `_matrix/client/unstable/add_threepid/msisdn/submit_token` endpoint. ([\matrix-org#13832](matrix-org#13832)) - Faster remote room joins: record _when_ we first partial-join to a room. ([\matrix-org#13892](matrix-org#13892)) - Support a `dir` parameter on the `/relations` endpoint per [MSC3715](matrix-org/matrix-spec-proposals#3715). ([\matrix-org#13920](matrix-org#13920)) - Ask mail servers receiving emails from Synapse to not send automatic replies (e.g. out-of-office responses). ([\matrix-org#13957](matrix-org#13957)) Bugfixes -------- - Send push notifications for invites received over federation. ([\matrix-org#13719](matrix-org#13719), [\matrix-org#14014](matrix-org#14014)) - Fix a long-standing bug where typing events would be accepted from remote servers not present in a room. Also fix a bug where incoming typing events would cause other incoming events to get stuck during a fast join. ([\matrix-org#13830](matrix-org#13830)) - Fix a bug introduced in Synapse v1.53.0 where the experimental implementation of [MSC3715](matrix-org/matrix-spec-proposals#3715) would give incorrect results when paginating forward. ([\matrix-org#13840](matrix-org#13840)) - Fix access token leak to logs from proxy agent. ([\matrix-org#13855](matrix-org#13855)) - Fix `have_seen_event` cache not being invalidated after we persist an event which causes inefficiency effects like extra `/state` federation calls. ([\matrix-org#13863](matrix-org#13863)) - Faster room joins: Fix a bug introduced in 1.66.0 where an error would be logged when syncing after joining a room. ([\matrix-org#13872](matrix-org#13872)) - Fix a bug introduced in 1.66.0 where some required fields in the pushrules sent to clients were not present anymore. Contributed by Nico. ([\matrix-org#13904](matrix-org#13904)) - Fix packaging to include `Cargo.lock` in `sdist`. ([\matrix-org#13909](matrix-org#13909)) - Fix a long-standing bug where device updates could cause delays sending out to-device messages over federation. ([\matrix-org#13922](matrix-org#13922)) - Fix a bug introduced in v1.68.0 where Synapse would require `setuptools_rust` at runtime, even though the package is only required at build time. ([\matrix-org#13952](matrix-org#13952)) - Fix a long-standing bug where `POST /_matrix/client/v3/keys/query` requests could result in excessively large SQL queries. ([\matrix-org#13956](matrix-org#13956)) - Fix a performance regression in the `get_users_in_room` database query. Introduced in v1.67.0. ([\matrix-org#13972](matrix-org#13972)) - Fix a bug introduced in v1.68.0 bug where Rust extension wasn't built in `release` mode when using `poetry install`. ([\matrix-org#14009](matrix-org#14009)) - Do not return an unspecified `original_event` field when using the stable `/relations` endpoint. Introduced in Synapse v1.57.0. ([\matrix-org#14025](matrix-org#14025)) - Correctly handle a race with device lists when a remote user leaves during a partial join. ([\matrix-org#13885](matrix-org#13885)) - Correctly handle sending local device list updates to remote servers during a partial join. ([\matrix-org#13934](matrix-org#13934)) Improved Documentation ---------------------- - Add `worker_main_http_uri` for the worker generator bash script. ([\matrix-org#13772](matrix-org#13772)) - Update URL for the NixOS module for Synapse. ([\matrix-org#13818](matrix-org#13818)) - Fix a mistake in sso_mapping_providers.md: `map_user_attributes` is expected to return `display_name`, not `displayname`. ([\matrix-org#13836](matrix-org#13836)) - Fix a cross-link from the registration admin API to the `registration_shared_secret` configuration documentation. ([\matrix-org#13870](matrix-org#13870)) - Update the man page for the `hash_password` script to correct the default number of bcrypt rounds performed. ([\matrix-org#13911](matrix-org#13911), [\matrix-org#13930](matrix-org#13930)) - Emphasize the right reasons when to use `(room_id, event_id)` in a database schema. ([\matrix-org#13915](matrix-org#13915)) - Add instruction to contributing guide for running unit tests in parallel. Contributed by @ashfame. ([\matrix-org#13928](matrix-org#13928)) - Clarify that the `auto_join_rooms` config option can also be used with Space aliases. ([\matrix-org#13931](matrix-org#13931)) - Add some cross references to worker documentation. ([\matrix-org#13974](matrix-org#13974)) - Linkify urls in config documentation. ([\matrix-org#14003](matrix-org#14003)) Deprecations and Removals ------------------------- - Remove the `complete_sso_login` method from the Module API which was deprecated in Synapse 1.13.0. ([\matrix-org#13843](matrix-org#13843)) - Announce that legacy metric names are deprecated, will be turned off by default in Synapse v1.71.0 and removed altogether in Synapse v1.73.0. See the upgrade notes for more information. ([\matrix-org#14024](matrix-org#14024)) Internal Changes ---------------- - Speed up creation of DM rooms. ([\matrix-org#13487](matrix-org#13487), [\matrix-org#13800](matrix-org#13800)) - Port push rules to using Rust. ([\matrix-org#13768](matrix-org#13768), [\matrix-org#13838](matrix-org#13838), [\matrix-org#13889](matrix-org#13889)) - Optimise get rooms for user calls. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13787](matrix-org#13787)) - Update the script which makes full schema dumps. ([\matrix-org#13792](matrix-org#13792)) - Use shared methods for cache invalidation when persisting events, remove duplicate codepaths. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13796](matrix-org#13796)) - Improve the `synapse.api.auth.Auth` mock used in unit tests. ([\matrix-org#13809](matrix-org#13809)) - Faster Remote Room Joins: tell remote homeservers that we are unable to authorise them if they query a room which has partial state on our server. ([\matrix-org#13823](matrix-org#13823)) - Carry IdP Session IDs through user-mapping sessions. ([\matrix-org#13839](matrix-org#13839)) - Fix the release script not publishing binary wheels. ([\matrix-org#13850](matrix-org#13850)) - Raise issue if complement fails with latest deps. ([\matrix-org#13859](matrix-org#13859)) - Correct the comments in the complement dockerfile. ([\matrix-org#13867](matrix-org#13867)) - Create a new snapshot of the database schema. ([\matrix-org#13873](matrix-org#13873)) - Faster room joins: Send device list updates to most servers in rooms with partial state. ([\matrix-org#13874](matrix-org#13874), [\matrix-org#14013](matrix-org#14013)) - Add comments to the Prometheus recording rules to make it clear which set of rules you need for Grafana or Prometheus Console. ([\matrix-org#13876](matrix-org#13876)) - Only pull relevant backfill points from the database based on the current depth and limit (instead of all) every time we want to `/backfill`. ([\matrix-org#13879](matrix-org#13879)) - Faster room joins: Avoid waiting for full state when processing `/keys/changes` requests. ([\matrix-org#13888](matrix-org#13888)) - Improve backfill robustness by trying more servers when we get a `4xx` error back. ([\matrix-org#13890](matrix-org#13890)) - Fix mypy errors with canonicaljson 1.6.3. ([\matrix-org#13905](matrix-org#13905)) - Faster remote room joins: correctly handle remote device list updates during a partial join. ([\matrix-org#13913](matrix-org#13913)) - Complement image: propagate SIGTERM to all workers. ([\matrix-org#13914](matrix-org#13914)) - Update an innaccurate comment in Synapse's upsert database helper. ([\matrix-org#13924](matrix-org#13924)) - Update mypy (0.950 -> 0.981) and mypy-zope (0.3.7 -> 0.3.11). ([\matrix-org#13925](matrix-org#13925), [\matrix-org#13993](matrix-org#13993)) - Use dedicated `get_local_users_in_room(room_id)` function to find local users when calculating users to copy over during a room upgrade. ([\matrix-org#13960](matrix-org#13960)) - Refactor language in user directory `_track_user_joined_room` code to make it more clear that we use both local and remote users. ([\matrix-org#13966](matrix-org#13966)) - Revert catch-all exceptions being recorded as event pull attempt failures (only handle what we know about). ([\matrix-org#13969](matrix-org#13969)) - Speed up calculating push actions in large rooms. ([\matrix-org#13973](matrix-org#13973), [\matrix-org#13992](matrix-org#13992)) - Enable update notifications from Github's dependabot. ([\matrix-org#13976](matrix-org#13976)) - Prototype a workflow to automatically add changelogs to dependabot PRs. ([\matrix-org#13998](matrix-org#13998), [\matrix-org#14011](matrix-org#14011), [\matrix-org#14017](matrix-org#14017), [\matrix-org#14021](matrix-org#14021), [\matrix-org#14027](matrix-org#14027)) - Fix type annotations to be compatible with new annotations in development versions of twisted. ([\matrix-org#14012](matrix-org#14012)) - Clear out stale entries in `event_push_actions_staging` table. ([\matrix-org#14020](matrix-org#14020)) - Bump versions of GitHub actions. ([\matrix-org#13978](matrix-org#13978), [\matrix-org#13979](matrix-org#13979), [\matrix-org#13980](matrix-org#13980), [\matrix-org#13982](matrix-org#13982), [\matrix-org#14015](matrix-org#14015), [\matrix-org#14019](matrix-org#14019), [\matrix-org#14022](matrix-org#14022), [\matrix-org#14023](matrix-org#14023)) # -----BEGIN PGP SIGNATURE----- # # iQFEBAABCgAuFiEEBTGR3/RnAzBGUif3pULk7RsPrAkFAmNNMOMQHGVyaWtAbWF0 # cml4Lm9yZwAKCRClQuTtGw+sCVnjB/9jpJRVnicteEpDfVX9iLo2qfIfcO/GhUJK # pJhv4yuY9whAldvJpmNw2f9tfUbAMcvrjlFvNrjihWmXcAGFazC6i3fNBjPgZW2e # Sxsuuy8xc9X/OqH2EUpHtNZQX3FfSbdBS93Z62ZO3R8tEbCQvjw6FXBdjjjf5uLO # y5Lsx94+41FJYOhs1Kt4fN92B9WMACR6e/O1YcsDjIXsoZI3uqO1h8filbQIZee7 # DTATE7eIPtShs2Ezaaeuc7tZGVDyPvgWIbuxuT6OGx20zmuChYJgIcVaD1me4UzJ # i9bVigtpYN0eUxuWnjLf7YC6Ys/Y9wZ7/lhdgaBwdbQKEJdpi+S4 # =JWaO # -----END PGP SIGNATURE----- # gpg: Signature made Mon Oct 17 11:39:31 2022 BST # gpg: using RSA key 053191DFF4670330465227F7A542E4ED1B0FAC09 # gpg: issuer "erik@matrix.org" # gpg: Can't check signature: No public key # Conflicts: # docker/Dockerfile # pyproject.toml # synapse/_scripts/update_synapse_database.py # synapse/handlers/message.py # synapse/handlers/receipts.py # synapse/logging/context.py # synapse/push/baserules.py # synapse/push/bulk_push_rule_evaluator.py # synapse/push/push_rule_evaluator.py # synapse/replication/http/send_event.py # synapse/rest/admin/users.py # synapse/rest/client/read_marker.py # synapse/rest/client/receipts.py # synapse/rest/client/room.py # synapse/storage/_base.py # synapse/storage/databases/main/__init__.py # synapse/storage/databases/main/cache.py # synapse/storage/databases/main/events.py # synapse/storage/databases/main/receipts.py # tests/push/test_push_rule_evaluator.py

erikjohnston and others added 18 commits September 9, 2022 15:48

Implement push rules as Rust native types.

c2d0c27

Remove python baserules

dcea8d8

Newsfile

62caed9

Update rust/src/push/mod.rs

071953f

Co-authored-by: David Robertson <davidr@element.io>

Merge remote-tracking branch 'origin/develop' into erikj/rust_push_rules

6605754

Code review

4d97349

Apply suggestions from code review

e828b34

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

Apply suggestions from code review

47a3dcf

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

Apply suggestions from code review

68689fa

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

Remove newsfile

a109a77

Remove old todo

dbeb21b

Swap actions

0e10bb3

Add back module docstring

1e05843

Merge remote-tracking branch 'origin/develop' into erikj/rust_push_rules

2945dc0

Remove debug logging

415bbd6

Handle unrecognized actions

86515a9

Correctly handle unknwon conditions

b57be51

Lint

a691a2d

erikjohnston commented Sep 19, 2022

View reviewed changes

erikjohnston added 2 commits September 19, 2022 12:25

Implement push rule evaluation in Rust

b730876

Ignore unknown conditions to fix unit tests

d62a7ec

erikjohnston force-pushed the erikj/rust_push_eval2 branch from 7528928 to d62a7ec Compare September 19, 2022 11:25

erikjohnston marked this pull request as ready for review September 19, 2022 11:35

erikjohnston requested a review from a team as a code owner September 19, 2022 11:35

erikjohnston commented Sep 19, 2022

View reviewed changes

MatMaul reviewed Sep 20, 2022

View reviewed changes

Update rust/src/push/evaluator.rs

65f55a2

Co-authored-by: Mathieu Velten <mathieuv@matrix.org>

MatMaul reviewed Sep 20, 2022

View reviewed changes

rust/src/push/evaluator.rs Show resolved Hide resolved

erikjohnston added 4 commits September 24, 2022 10:37

Merge remote-tracking branch 'origin/develop' into erikj/rust_push_eval2

4da5b04

Speed up

e6d9341

Add a Matcher type

bef2298

Tidy up

c51bf1e

erikjohnston added 4 commits September 25, 2022 14:54

Remove unnecessary optimisation

307e77a

Comment

f760ada

clippy

9df6d37

Copyright headers

e4cd075

erikjohnston requested a review from MatMaul September 27, 2022 08:01

erikjohnston requested a review from a team September 28, 2022 09:52

clokep reviewed Sep 28, 2022

View reviewed changes

rust/Cargo.toml Outdated Show resolved Hide resolved

clokep requested a review from a team September 28, 2022 14:40

erikjohnston and others added 2 commits September 28, 2022 17:29

Update rust/Cargo.toml

0035f14

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

Merge remote-tracking branch 'origin/develop' into erikj/rust_push_eval2

a503245

reivilibre suggested changes Sep 29, 2022

View reviewed changes

erikjohnston and others added 3 commits September 29, 2022 14:03

Review comments

1034e7f

Fix merge

de494f2

Apply suggestions from code review

23b307d

Co-authored-by: reivilibre <oliverw@matrix.org>

erikjohnston requested a review from reivilibre September 29, 2022 13:12

Fix test

09b728d

reivilibre approved these changes Sep 29, 2022

View reviewed changes

erikjohnston merged commit ebd9e2d into develop Sep 29, 2022

erikjohnston deleted the erikj/rust_push_eval2 branch September 29, 2022 15:12

DMRobertson mentioned this pull request Oct 5, 2022

Cannot send new events into rooms with floating-point (and stringy?) power levels #14060

Closed

clokep mentioned this pull request Jan 30, 2023

Handle malformed values of notification.room in power level events #14942

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement push rule evaluation in Rust. #13838

Implement push rule evaluation in Rust. #13838

erikjohnston commented Sep 19, 2022

erikjohnston Sep 19, 2022

clokep Sep 27, 2022

erikjohnston Sep 27, 2022

clokep Sep 27, 2022 •

edited

Loading

erikjohnston Sep 27, 2022

erikjohnston Sep 19, 2022

erikjohnston Sep 19, 2022

erikjohnston Sep 19, 2022

MatMaul Sep 20, 2022

erikjohnston Sep 20, 2022

erikjohnston Sep 20, 2022

erikjohnston Sep 24, 2022

erikjohnston commented Sep 24, 2022

erikjohnston commented Sep 27, 2022

reivilibre left a comment

reivilibre Sep 29, 2022

erikjohnston Sep 29, 2022

reivilibre Sep 29, 2022 •

edited

Loading

erikjohnston Sep 29, 2022

erikjohnston Sep 29, 2022

reivilibre Sep 29, 2022

reivilibre left a comment

erikjohnston commented Sep 29, 2022

	else:
	# XXX This looks incorrect -- we have reached an unknown condition
	# kind and are unconditionally returning that it matches. Note
	# that it seems possible to provide a condition to the /pushrules
	# endpoint with an unknown kind, see _rule_tuple_from_request_object.
	return True

		@@ -56,6 +56,39 @@
		)


		def tweaks_for_actions(actions: List[Union[str, Dict]]) -> Dict[str, Any]:

Implement push rule evaluation in Rust. #13838

Implement push rule evaluation in Rust. #13838

Conversation

erikjohnston commented Sep 19, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clokep Sep 27, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erikjohnston commented Sep 24, 2022

erikjohnston commented Sep 27, 2022

reivilibre left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reivilibre Sep 29, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reivilibre left a comment

Choose a reason for hiding this comment

erikjohnston commented Sep 29, 2022

clokep Sep 27, 2022 •

edited

Loading

reivilibre Sep 29, 2022 •

edited

Loading