This repository has been archived by the owner on Apr 26, 2024. It is now read-only.
Increase perf of handling presence when joining large rooms. #9916
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We ended up doing a *lot* of duplicate work, and e.g. ended up doing n^2 worth of `is_mine_id(..)` checks across all joined users.
richvdh
approved these changes
Apr 30, 2021
synapse/handlers/presence.py
Outdated
Comment on lines
1259
to
1265
| prev_local_users = set() | ||
| prev_remote_hosts = set() | ||
| for user_id in prev_users: | ||
| if self.is_mine_id(user_id): | ||
| prev_local_users.add(user_id) | ||
| else: | ||
| prev_remote_hosts.add(get_domain_from_id(user_id)) |
There was a problem hiding this comment.
I wonder if a general "partition a user list by homeserver" function would be generally useful. just a thought, no need to do anything about it.
There was a problem hiding this comment.
Yeah, that would probably make a lot of sense
|
|
||
| # Get all the users who were already in the room, by fetching the | ||
| # current users in the room and removing the newly joined users. | ||
| users = await self.store.get_users_in_room(room_id) |
There was a problem hiding this comment.
can we rely on all deltas being for the same room id? and can we rely on there being at least one delta (so that room_id is set)?
There was a problem hiding this comment.
Oh, that is a good point. I don't think we can rely on there only being one room per call (though that is the common path).
richvdh
approved these changes
May 4, 2021
aaronraimist
added a commit
to aaronraimist/synapse
that referenced
this pull request
May 19, 2021
Synapse 1.34.0 (2021-05-17) =========================== This release deprecates the `room_invite_state_types` configuration setting. See the [upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.34.0/UPGRADE.rst#upgrading-to-v1340) for instructions on updating your configuration file to use the new `room_prejoin_state` setting. This release also deprecates the `POST /_synapse/admin/v1/rooms/<room_id>/delete` admin API route. Server administrators are encouraged to update their scripts to use the new `DELETE /_synapse/admin/v1/rooms/<room_id>` route instead. No significant changes since v1.34.0rc1. Synapse 1.34.0rc1 (2021-05-12) ============================== Features -------- - Add experimental option to track memory usage of the caches. ([\matrix-org#9881](matrix-org#9881)) - Add support for `DELETE /_synapse/admin/v1/rooms/<room_id>`. ([\matrix-org#9889](matrix-org#9889)) - Add limits to how often Synapse will GC, ensuring that large servers do not end up GC thrashing if `gc_thresholds` has not been correctly set. ([\matrix-org#9902](matrix-org#9902)) - Improve performance of sending events for worker-based deployments using Redis. ([\matrix-org#9905](matrix-org#9905), [\matrix-org#9950](matrix-org#9950), [\matrix-org#9951](matrix-org#9951)) - Improve performance after joining a large room when presence is enabled. ([\matrix-org#9910](matrix-org#9910), [\matrix-org#9916](matrix-org#9916)) - Support stable identifiers for [MSC1772](matrix-org/matrix-spec-proposals#1772) Spaces. `m.space.child` events will now be taken into account when populating the experimental spaces summary response. Please see [the upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.34.0/UPGRADE.rst#upgrading-to-v1340) if you have customised `room_invite_state_types` in your configuration. ([\matrix-org#9915](matrix-org#9915), [\matrix-org#9966](matrix-org#9966)) - Improve performance of backfilling in large rooms. ([\matrix-org#9935](matrix-org#9935)) - Add a config option to allow you to prevent device display names from being shared over federation. Contributed by @aaronraimist. ([\matrix-org#9945](matrix-org#9945)) - Update support for [MSC2946](matrix-org/matrix-spec-proposals#2946): Spaces Summary. ([\matrix-org#9947](matrix-org#9947), [\matrix-org#9954](matrix-org#9954)) Bugfixes -------- - Fix a bug introduced in v1.32.0 where the associated connection was improperly logged for SQL logging statements. ([\matrix-org#9895](matrix-org#9895)) - Correct the type hint for the `user_may_create_room_alias` method of spam checkers. It is provided a `RoomAlias`, not a `str`. ([\matrix-org#9896](matrix-org#9896)) - Fix bug where user directory could get out of sync if room visibility and membership changed in quick succession. ([\matrix-org#9910](matrix-org#9910)) - Include the `origin_server_ts` property in the experimental [MSC2946](matrix-org/matrix-spec-proposals#2946) support to allow clients to properly sort rooms. ([\matrix-org#9928](matrix-org#9928)) - Fix bugs introduced in v1.23.0 which made the PostgreSQL port script fail when run with a newly-created SQLite database. ([\matrix-org#9930](matrix-org#9930)) - Fix a bug introduced in Synapse 1.29.0 which caused `m.room_key_request` to-device messages sent from one user to another to be dropped. ([\matrix-org#9961](matrix-org#9961), [\matrix-org#9965](matrix-org#9965)) - Fix a bug introduced in v1.27.0 preventing users and appservices exempt from ratelimiting from creating rooms with many invitees. ([\matrix-org#9968](matrix-org#9968)) Updates to the Docker image --------------------------- - Add `startup_delay` to docker healthcheck to reduce waiting time for coming online and update the documentation with extra options. Contributed by @maquis196. ([\matrix-org#9913](matrix-org#9913)) Improved Documentation ---------------------- - Add `port` argument to the Postgres database sample config section. ([\matrix-org#9911](matrix-org#9911)) Deprecations and Removals ------------------------- - Mark as deprecated `POST /_synapse/admin/v1/rooms/<room_id>/delete`. ([\matrix-org#9889](matrix-org#9889)) Internal Changes ---------------- - Reduce the length of Synapse's access tokens. ([\matrix-org#5588](matrix-org#5588)) - Export jemalloc stats to Prometheus if it is being used. ([\matrix-org#9882](matrix-org#9882)) - Add type hints to presence handler. ([\matrix-org#9885](matrix-org#9885)) - Reduce memory usage of the LRU caches. ([\matrix-org#9886](matrix-org#9886)) - Add type hints to the `synapse.handlers` module. ([\matrix-org#9896](matrix-org#9896)) - Time response time for external cache requests. ([\matrix-org#9904](matrix-org#9904)) - Minor fixes to the `make_full_schema.sh` script. ([\matrix-org#9931](matrix-org#9931)) - Move database schema files into a common directory. ([\matrix-org#9932](matrix-org#9932)) - Add debug logging for lost/delayed to-device messages. ([\matrix-org#9959](matrix-org#9959))
vy-let
added a commit
to vy-let/synapse
that referenced
this pull request
May 31, 2021
Synapse 1.34.0 (2021-05-17) =========================== This release deprecates the `room_invite_state_types` configuration setting. See the [upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.34.0/UPGRADE.rst#upgrading-to-v1340) for instructions on updating your configuration file to use the new `room_prejoin_state` setting. This release also deprecates the `POST /_synapse/admin/v1/rooms/<room_id>/delete` admin API route. Server administrators are encouraged to update their scripts to use the new `DELETE /_synapse/admin/v1/rooms/<room_id>` route instead. No significant changes since v1.34.0rc1. Synapse 1.34.0rc1 (2021-05-12) ============================== Features -------- - Add experimental option to track memory usage of the caches. ([\matrix-org#9881](matrix-org#9881)) - Add support for `DELETE /_synapse/admin/v1/rooms/<room_id>`. ([\matrix-org#9889](matrix-org#9889)) - Add limits to how often Synapse will GC, ensuring that large servers do not end up GC thrashing if `gc_thresholds` has not been correctly set. ([\matrix-org#9902](matrix-org#9902)) - Improve performance of sending events for worker-based deployments using Redis. ([\matrix-org#9905](matrix-org#9905), [\matrix-org#9950](matrix-org#9950), [\matrix-org#9951](matrix-org#9951)) - Improve performance after joining a large room when presence is enabled. ([\matrix-org#9910](matrix-org#9910), [\matrix-org#9916](matrix-org#9916)) - Support stable identifiers for [MSC1772](matrix-org/matrix-spec-proposals#1772) Spaces. `m.space.child` events will now be taken into account when populating the experimental spaces summary response. Please see [the upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.34.0/UPGRADE.rst#upgrading-to-v1340) if you have customised `room_invite_state_types` in your configuration. ([\matrix-org#9915](matrix-org#9915), [\matrix-org#9966](matrix-org#9966)) - Improve performance of backfilling in large rooms. ([\matrix-org#9935](matrix-org#9935)) - Add a config option to allow you to prevent device display names from being shared over federation. Contributed by @aaronraimist. ([\matrix-org#9945](matrix-org#9945)) - Update support for [MSC2946](matrix-org/matrix-spec-proposals#2946): Spaces Summary. ([\matrix-org#9947](matrix-org#9947), [\matrix-org#9954](matrix-org#9954)) Bugfixes -------- - Fix a bug introduced in v1.32.0 where the associated connection was improperly logged for SQL logging statements. ([\matrix-org#9895](matrix-org#9895)) - Correct the type hint for the `user_may_create_room_alias` method of spam checkers. It is provided a `RoomAlias`, not a `str`. ([\matrix-org#9896](matrix-org#9896)) - Fix bug where user directory could get out of sync if room visibility and membership changed in quick succession. ([\matrix-org#9910](matrix-org#9910)) - Include the `origin_server_ts` property in the experimental [MSC2946](matrix-org/matrix-spec-proposals#2946) support to allow clients to properly sort rooms. ([\matrix-org#9928](matrix-org#9928)) - Fix bugs introduced in v1.23.0 which made the PostgreSQL port script fail when run with a newly-created SQLite database. ([\matrix-org#9930](matrix-org#9930)) - Fix a bug introduced in Synapse 1.29.0 which caused `m.room_key_request` to-device messages sent from one user to another to be dropped. ([\matrix-org#9961](matrix-org#9961), [\matrix-org#9965](matrix-org#9965)) - Fix a bug introduced in v1.27.0 preventing users and appservices exempt from ratelimiting from creating rooms with many invitees. ([\matrix-org#9968](matrix-org#9968)) Updates to the Docker image --------------------------- - Add `startup_delay` to docker healthcheck to reduce waiting time for coming online and update the documentation with extra options. Contributed by @maquis196. ([\matrix-org#9913](matrix-org#9913)) Improved Documentation ---------------------- - Add `port` argument to the Postgres database sample config section. ([\matrix-org#9911](matrix-org#9911)) Deprecations and Removals ------------------------- - Mark as deprecated `POST /_synapse/admin/v1/rooms/<room_id>/delete`. ([\matrix-org#9889](matrix-org#9889)) Internal Changes ---------------- - Reduce the length of Synapse's access tokens. ([\matrix-org#5588](matrix-org#5588)) - Export jemalloc stats to Prometheus if it is being used. ([\matrix-org#9882](matrix-org#9882)) - Add type hints to presence handler. ([\matrix-org#9885](matrix-org#9885)) - Reduce memory usage of the LRU caches. ([\matrix-org#9886](matrix-org#9886)) - Add type hints to the `synapse.handlers` module. ([\matrix-org#9896](matrix-org#9896)) - Time response time for external cache requests. ([\matrix-org#9904](matrix-org#9904)) - Minor fixes to the `make_full_schema.sh` script. ([\matrix-org#9931](matrix-org#9931)) - Move database schema files into a common directory. ([\matrix-org#9932](matrix-org#9932)) - Add debug logging for lost/delayed to-device messages. ([\matrix-org#9959](matrix-org#9959))
babolivier
added a commit
to matrix-org/synapse-dinsic
that referenced
this pull request
Sep 1, 2021
Synapse 1.34.0 (2021-05-17) =========================== This release deprecates the `room_invite_state_types` configuration setting. See the [upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.34.0/UPGRADE.rst#upgrading-to-v1340) for instructions on updating your configuration file to use the new `room_prejoin_state` setting. This release also deprecates the `POST /_synapse/admin/v1/rooms/<room_id>/delete` admin API route. Server administrators are encouraged to update their scripts to use the new `DELETE /_synapse/admin/v1/rooms/<room_id>` route instead. No significant changes since v1.34.0rc1. Synapse 1.34.0rc1 (2021-05-12) ============================== Features -------- - Add experimental option to track memory usage of the caches. ([\#9881](matrix-org/synapse#9881)) - Add support for `DELETE /_synapse/admin/v1/rooms/<room_id>`. ([\#9889](matrix-org/synapse#9889)) - Add limits to how often Synapse will GC, ensuring that large servers do not end up GC thrashing if `gc_thresholds` has not been correctly set. ([\#9902](matrix-org/synapse#9902)) - Improve performance of sending events for worker-based deployments using Redis. ([\#9905](matrix-org/synapse#9905), [\#9950](matrix-org/synapse#9950), [\#9951](matrix-org/synapse#9951)) - Improve performance after joining a large room when presence is enabled. ([\#9910](matrix-org/synapse#9910), [\#9916](matrix-org/synapse#9916)) - Support stable identifiers for [MSC1772](matrix-org/matrix-spec-proposals#1772) Spaces. `m.space.child` events will now be taken into account when populating the experimental spaces summary response. Please see [the upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.34.0/UPGRADE.rst#upgrading-to-v1340) if you have customised `room_invite_state_types` in your configuration. ([\#9915](matrix-org/synapse#9915), [\#9966](matrix-org/synapse#9966)) - Improve performance of backfilling in large rooms. ([\#9935](matrix-org/synapse#9935)) - Add a config option to allow you to prevent device display names from being shared over federation. Contributed by @aaronraimist. ([\#9945](matrix-org/synapse#9945)) - Update support for [MSC2946](matrix-org/matrix-spec-proposals#2946): Spaces Summary. ([\#9947](matrix-org/synapse#9947), [\#9954](matrix-org/synapse#9954)) Bugfixes -------- - Fix a bug introduced in v1.32.0 where the associated connection was improperly logged for SQL logging statements. ([\#9895](matrix-org/synapse#9895)) - Correct the type hint for the `user_may_create_room_alias` method of spam checkers. It is provided a `RoomAlias`, not a `str`. ([\#9896](matrix-org/synapse#9896)) - Fix bug where user directory could get out of sync if room visibility and membership changed in quick succession. ([\#9910](matrix-org/synapse#9910)) - Include the `origin_server_ts` property in the experimental [MSC2946](matrix-org/matrix-spec-proposals#2946) support to allow clients to properly sort rooms. ([\#9928](matrix-org/synapse#9928)) - Fix bugs introduced in v1.23.0 which made the PostgreSQL port script fail when run with a newly-created SQLite database. ([\#9930](matrix-org/synapse#9930)) - Fix a bug introduced in Synapse 1.29.0 which caused `m.room_key_request` to-device messages sent from one user to another to be dropped. ([\#9961](matrix-org/synapse#9961), [\#9965](matrix-org/synapse#9965)) - Fix a bug introduced in v1.27.0 preventing users and appservices exempt from ratelimiting from creating rooms with many invitees. ([\#9968](matrix-org/synapse#9968)) Updates to the Docker image --------------------------- - Add `startup_delay` to docker healthcheck to reduce waiting time for coming online and update the documentation with extra options. Contributed by @maquis196. ([\#9913](matrix-org/synapse#9913)) Improved Documentation ---------------------- - Add `port` argument to the Postgres database sample config section. ([\#9911](matrix-org/synapse#9911)) Deprecations and Removals ------------------------- - Mark as deprecated `POST /_synapse/admin/v1/rooms/<room_id>/delete`. ([\#9889](matrix-org/synapse#9889)) Internal Changes ---------------- - Reduce the length of Synapse's access tokens. ([\#5588](matrix-org/synapse#5588)) - Export jemalloc stats to Prometheus if it is being used. ([\#9882](matrix-org/synapse#9882)) - Add type hints to presence handler. ([\#9885](matrix-org/synapse#9885)) - Reduce memory usage of the LRU caches. ([\#9886](matrix-org/synapse#9886)) - Add type hints to the `synapse.handlers` module. ([\#9896](matrix-org/synapse#9896)) - Time response time for external cache requests. ([\#9904](matrix-org/synapse#9904)) - Minor fixes to the `make_full_schema.sh` script. ([\#9931](matrix-org/synapse#9931)) - Move database schema files into a common directory. ([\#9932](matrix-org/synapse#9932)) - Add debug logging for lost/delayed to-device messages. ([\#9959](matrix-org/synapse#9959))
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We ended up doing a lot of duplicate work, and e.g. ended up doing n^2
worth of
is_mine_id(..)checks across all joined users.Based on #9910