Make DictionaryCache have better expiry properties #13292

erikjohnston · 2022-07-15T14:26:24Z

The DictionaryCache is a cache designed for caching lookups into mappings, mainly for looking up state groups. It can handle fetching both subsets of the dict, or the entire thing.

Before, this worked by storing a DictionaryEntry in an LruCache, updating the stored dictionary whenever we pulled more entries for the DB. The disadvantage of this method is that we'd never expire subsets of the dictionary that we didn't query, so long as that other parts of the dictionary were being queried. For example, if we repeatedly fetch the create event of a room, but at some point ask for the full state of the room, we'd never then drop the full state from the cache, even though we only care about the create event.

This leads to severely bloated state group caches.

Instead, this PR splits up the entries it stores in the LruCache into two distinct types:

Individual entries in the dictionary, under a (key, dict_key) (e.g. for state group caches this would be (state_group, (type, state_key))). We set the LruCache to use the TreeCache to enable us to query/invalidate all entries for a given key.
The full dict for a given key, under (key, FullDict.Key).

This allow us to expire the individual entries and the full dict separately.

tests/storage/test_state.py

richvdh

I've started reviewing this, but I'm confused and don't want to spend any longer on it right now.

synapse/util/caches/dictionary_cache.py

synapse/util/caches/lrucache.py

synapse/util/caches/dictionary_cache.py

synapse/util/caches/treecache.py

tests/storage/test_state.py

synapse/util/caches/dictionary_cache.py

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

erikjohnston · 2022-07-19T13:18:35Z

LMK if the clarifications aren't enough

richvdh

well, I've stared at it all morning, and now understand how it works. A bunch of suggestions, mostly for clarification.

Any chance of testing any of this? We have a DictCacheTestCase which we might be able to extend?

synapse/util/caches/dictionary_cache.py

richvdh · 2022-07-20T11:07:25Z

synapse/util/caches/dictionary_cache.py

@@ -149,20 +286,27 @@ def update(
 # Only update the cache if the caches sequence number matches the
 # number that the cache had before the SELECT was started (SYN-369)
 if fetched_keys is None:
- self._insert(key, value, set())
+ self.cache[(key, _FullCacheKey.KEY)] = value


do we not need to update (or at least invalidate) entries for individual dict keys here as well?

I don't think so, if the value in the DB has changed then we should have previously called invalidate(..) to delete the stale entries? We could easily drop any existing entries, but I don't think its necessary?

fair enough. That seems like an assumption that we should document in the docstring though.

(Also: shouldn't this method be called set or put or insert or something? Maybe a problem for another time though.)

(Also: shouldn't this method be called set or put or insert or something? Maybe a problem for another time though.)

Yup, will try and put up a separate PR to do some of these fixes if they're small.

synapse/util/caches/dictionary_cache.py

richvdh · 2022-07-20T11:16:13Z

synapse/storage/databases/state/store.py

+ # If we are asked explicitly for a subset of keys, we only ask for those
+ # from the cache. This ensures that the `DictionaryCache` can make
+ # better decisions about what to cache and what to expire.
+ dict_keys = None
+ if not state_filter.has_wildcards():
+ dict_keys = state_filter.concrete_types()
+
+ cache_entry = cache.get(group, dict_keys=dict_keys)


ideally we'd separate this to a different PR. It's not directly related to the changes to DictionaryCache.

richvdh · 2022-07-21T12:04:37Z

synapse/util/caches/dictionary_cache.py

 Returns:
 DictionaryEntry


this doesn't document what happens if dict_keys is None, but we don't have the full dict cached. Could it, please?

... please?

I think it returns some fixed shape of DictionaryEntry ?

Ah, I'd put some docs under the dict_keys arg (and there is some in DictionaryEntry), but have also added some in the return value too.

synapse/util/caches/dictionary_cache.py

richvdh · 2022-07-21T12:11:01Z

synapse/util/caches/dictionary_cache.py

- self.cache.pop(key, None)
+
+ # We want to drop all information about the dict for the given key, so
+ # we use `del_multi` to delete it all in one go.
+ #
+ # We ignore the type error here `del_mutli` accepts a truncated key
+ # (when the key type is a tuple).
+ self.cache.del_multi((key,)) # type: ignore[arg-type]


so how did this ever work before?

It wasn't a TreeCache before: it was just using a DictionaryEntry as the cache value, and keeping track of whether that was "full" or "partial".

It wasn't a TreeCache before:

oh gosh, I'd completely missed that.

richvdh · 2022-07-21T12:16:10Z

synapse/util/caches/dictionary_cache.py

@@ -149,20 +286,27 @@ def update(
 # Only update the cache if the caches sequence number matches the
 # number that the cache had before the SELECT was started (SYN-369)
 if fetched_keys is None:
- self._insert(key, value, set())
+ self.cache[(key, _FullCacheKey.KEY)] = value


fair enough. That seems like an assumption that we should document in the docstring though.

(Also: shouldn't this method be called set or put or insert or something? Maybe a problem for another time though.)

synapse/util/caches/lrucache.py

richvdh

looks generally good. still a few comments.

synapse/util/caches/treecache.py

tests/util/test_dict_cache.py

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

richvdh

lgtm other than the docs for the behaviour of .get

richvdh · 2022-07-21T13:54:31Z

synapse/util/caches/dictionary_cache.py

 Returns:
 DictionaryEntry


... please?

I think it returns some fixed shape of DictionaryEntry ?

synapse/util/caches/dictionary_cache.py

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

richvdh

🚢

erikjohnston · 2022-07-21T16:13:39Z

Woop woop! Thank you ❤️

@andrewdoh

Synapse 1.64.0 (2022-08-02) =========================== No significant changes since 1.64.0rc2. Deprecation Warning ------------------- Synapse v1.66.0 will remove the ability to delegate the tasks of verifying email address ownership, and password reset confirmation, to an identity server. If you require your homeserver to verify e-mail addresses or to support password resets via e-mail, please configure your homeserver with SMTP access so that it can send e-mails on its own behalf. [Consult the configuration documentation for more information.](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#email) Synapse 1.64.0rc2 (2022-07-29) ============================== This RC reintroduces support for `account_threepid_delegates.email`, which was removed in 1.64.0rc1. It remains deprecated and will be removed altogether in Synapse v1.66.0. ([\matrix-org#13406](matrix-org#13406)) Synapse 1.64.0rc1 (2022-07-26) ============================== This RC removed the ability to delegate the tasks of verifying email address ownership, and password reset confirmation, to an identity server. We have also stopped building `.deb` packages for Ubuntu 21.10 as it is no longer an active version of Ubuntu. Features -------- - Improve error messages when media thumbnails cannot be served. ([\matrix-org#13038](matrix-org#13038)) - Allow pagination from remote event after discovering it from [MSC3030](matrix-org/matrix-spec-proposals#3030) `/timestamp_to_event`. ([\matrix-org#13205](matrix-org#13205)) - Add a `room_type` field in the responses for the list room and room details admin APIs. Contributed by @andrewdoh. ([\matrix-org#13208](matrix-org#13208)) - Add support for room version 10. ([\matrix-org#13220](matrix-org#13220)) - Add per-room rate limiting for room joins. For each room, Synapse now monitors the rate of join events in that room, and throttles additional joins if that rate grows too large. ([\matrix-org#13253](matrix-org#13253), [\matrix-org#13254](matrix-org#13254), [\matrix-org#13255](matrix-org#13255), [\matrix-org#13276](matrix-org#13276)) - Support Implicit TLS (TLS without using a STARTTLS upgrade, typically on port 465) for sending emails, enabled by the new option `force_tls`. Contributed by Jan Schär. ([\matrix-org#13317](matrix-org#13317)) Bugfixes -------- - Fix a bug introduced in Synapse 1.15.0 where adding a user through the Synapse Admin API with a phone number would fail if the `enable_email_notifs` and `email_notifs_for_new_users` options were enabled. Contributed by @thomasweston12. ([\matrix-org#13263](matrix-org#13263)) - Fix a bug introduced in Synapse 1.40.0 where a user invited to a restricted room would be briefly unable to join. ([\matrix-org#13270](matrix-org#13270)) - Fix a long-standing bug where, in rare instances, Synapse could store the incorrect state for a room after a state resolution. ([\matrix-org#13278](matrix-org#13278)) - Fix a bug introduced in v1.18.0 where the `synapse_pushers` metric would overcount pushers when they are replaced. ([\matrix-org#13296](matrix-org#13296)) - Disable autocorrection and autocapitalisation on the username text field shown during registration when using SSO. ([\matrix-org#13350](matrix-org#13350)) - Update locked version of `frozendict` to 2.3.3, which has fixes for memory leaks affecting `/sync`. ([\matrix-org#13284](matrix-org#13284), [\matrix-org#13352](matrix-org#13352)) Improved Documentation ---------------------- - Provide an example of using the Admin API. Contributed by @jejo86. ([\matrix-org#13231](matrix-org#13231)) - Move the documentation for how URL previews work to the URL preview module. ([\matrix-org#13233](matrix-org#13233), [\matrix-org#13261](matrix-org#13261)) - Add another `contrib` script to help set up worker processes. Contributed by @villepeh. ([\matrix-org#13271](matrix-org#13271)) - Document that certain config options were added or changed in Synapse 1.62. Contributed by @behrmann. ([\matrix-org#13314](matrix-org#13314)) - Document the new `rc_invites.per_issuer` throttling option added in Synapse 1.63. ([\matrix-org#13333](matrix-org#13333)) - Mention that BuildKit is needed when building Docker images for tests. ([\matrix-org#13338](matrix-org#13338)) - Improve Caddy reverse proxy documentation. ([\matrix-org#13344](matrix-org#13344)) Deprecations and Removals ------------------------- - Drop tables that were formerly used for groups/communities. ([\matrix-org#12967](matrix-org#12967)) - Drop support for delegating email verification to an external server. ([\matrix-org#13192](matrix-org#13192)) - Drop support for calling `/_matrix/client/v3/account/3pid/bind` without an `id_access_token`, which was not permitted by the spec. Contributed by @Vetchu. ([\matrix-org#13239](matrix-org#13239)) - Stop building `.deb` packages for Ubuntu 21.10 (Impish Indri), which has reached end of life. ([\matrix-org#13326](matrix-org#13326)) Internal Changes ---------------- - Use lower transaction isolation level when purging rooms to avoid serialization errors. Contributed by Nick @ Beeper. ([\matrix-org#12942](matrix-org#12942)) - Remove code which incorrectly attempted to reconcile state with remote servers when processing incoming events. ([\matrix-org#12943](matrix-org#12943)) - Make the AS login method call `Auth.get_user_by_req` for checking the AS token. ([\matrix-org#13094](matrix-org#13094)) - Always use a version of canonicaljson that supports the C implementation of frozendict. ([\matrix-org#13172](matrix-org#13172)) - Add prometheus counters for ephemeral events and to device messages pushed to app services. Contributed by Brad @ Beeper. ([\matrix-org#13175](matrix-org#13175)) - Refactor receipts servlet logic to avoid duplicated code. ([\matrix-org#13198](matrix-org#13198)) - Preparation for database schema simplifications: populate `state_key` and `rejection_reason` for existing rows in the `events` table. ([\matrix-org#13215](matrix-org#13215)) - Remove unused database table `event_reference_hashes`. ([\matrix-org#13218](matrix-org#13218)) - Further reduce queries used sending events when creating new rooms. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13224](matrix-org#13224)) - Call the v2 identity service `/3pid/unbind` endpoint, rather than v1. Contributed by @Vetchu. ([\matrix-org#13240](matrix-org#13240)) - Use an asynchronous cache wrapper for the get event cache. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13242](matrix-org#13242), [\matrix-org#13308](matrix-org#13308)) - Optimise federation sender and appservice pusher event stream processing queries. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13251](matrix-org#13251)) - Log the stack when waiting for an entire room to be un-partial stated. ([\matrix-org#13257](matrix-org#13257)) - Fix spurious warning when fetching state after a missing prev event. ([\matrix-org#13258](matrix-org#13258)) - Clean-up tests for notifications. ([\matrix-org#13260](matrix-org#13260)) - Do not fail build if complement with workers fails. ([\matrix-org#13266](matrix-org#13266)) - Don't pull out state in `compute_event_context` for unconflicted state. ([\matrix-org#13267](matrix-org#13267), [\matrix-org#13274](matrix-org#13274)) - Reduce the rebuild time for the complement-synapse docker image. ([\matrix-org#13279](matrix-org#13279)) - Don't pull out the full state when creating an event. ([\matrix-org#13281](matrix-org#13281), [\matrix-org#13307](matrix-org#13307)) - Upgrade from Poetry 1.1.12 to 1.1.14, to fix bugs when locking packages. ([\matrix-org#13285](matrix-org#13285)) - Make `DictionaryCache` expire full entries if they haven't been queried in a while, even if specific keys have been queried recently. ([\matrix-org#13292](matrix-org#13292)) - Use `HTTPStatus` constants in place of literals in tests. ([\matrix-org#13297](matrix-org#13297)) - Improve performance of query `_get_subset_users_in_room_with_profiles`. ([\matrix-org#13299](matrix-org#13299)) - Up batch size of `bulk_get_push_rules` and `_get_joined_profiles_from_event_ids`. ([\matrix-org#13300](matrix-org#13300)) - Remove unnecessary `json.dumps` from tests. ([\matrix-org#13303](matrix-org#13303)) - Reduce memory usage of sending dummy events. ([\matrix-org#13310](matrix-org#13310)) - Prevent formatting changes of [matrix-org#3679](matrix-org#3679) from appearing in `git blame`. ([\matrix-org#13311](matrix-org#13311)) - Change `get_users_in_room` and `get_rooms_for_user` caches to enable pruning of old entries. ([\matrix-org#13313](matrix-org#13313)) - Validate federation destinations and log an error if a destination is invalid. ([\matrix-org#13318](matrix-org#13318)) - Fix `FederationClient.get_pdu()` returning events from the cache as `outliers` instead of original events we saw over federation. ([\matrix-org#13320](matrix-org#13320)) - Reduce memory usage of state caches. ([\matrix-org#13323](matrix-org#13323)) - Reduce the amount of state we store in the `state_cache`. ([\matrix-org#13324](matrix-org#13324)) - Add missing type hints to open tracing module. ([\matrix-org#13328](matrix-org#13328), [\matrix-org#13345](matrix-org#13345), [\matrix-org#13362](matrix-org#13362)) - Remove old base slaved store and de-duplicate cache ID generators. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13329](matrix-org#13329), [\matrix-org#13349](matrix-org#13349)) - When reporting metrics is enabled, use ~8x less data to describe DB transaction metrics. ([\matrix-org#13342](matrix-org#13342)) - Faster room joins: skip soft fail checks while Synapse only has partial room state, since the current membership of event senders may not be accurately known. ([\matrix-org#13354](matrix-org#13354))

@andrewdoh

Synapse 1.64.0 (2022-08-02) =========================== No significant changes since 1.64.0rc2. Deprecation Warning ------------------- Synapse v1.66.0 will remove the ability to delegate the tasks of verifying email address ownership, and password reset confirmation, to an identity server. If you require your homeserver to verify e-mail addresses or to support password resets via e-mail, please configure your homeserver with SMTP access so that it can send e-mails on its own behalf. [Consult the configuration documentation for more information.](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#email) Synapse 1.64.0rc2 (2022-07-29) ============================== This RC reintroduces support for `account_threepid_delegates.email`, which was removed in 1.64.0rc1. It remains deprecated and will be removed altogether in Synapse v1.66.0. ([\matrix-org#13406](matrix-org#13406)) Synapse 1.64.0rc1 (2022-07-26) ============================== This RC removed the ability to delegate the tasks of verifying email address ownership, and password reset confirmation, to an identity server. We have also stopped building `.deb` packages for Ubuntu 21.10 as it is no longer an active version of Ubuntu. Features -------- - Improve error messages when media thumbnails cannot be served. ([\matrix-org#13038](matrix-org#13038)) - Allow pagination from remote event after discovering it from [MSC3030](matrix-org/matrix-spec-proposals#3030) `/timestamp_to_event`. ([\matrix-org#13205](matrix-org#13205)) - Add a `room_type` field in the responses for the list room and room details admin APIs. Contributed by @andrewdoh. ([\matrix-org#13208](matrix-org#13208)) - Add support for room version 10. ([\matrix-org#13220](matrix-org#13220)) - Add per-room rate limiting for room joins. For each room, Synapse now monitors the rate of join events in that room, and throttles additional joins if that rate grows too large. ([\matrix-org#13253](matrix-org#13253), [\matrix-org#13254](matrix-org#13254), [\matrix-org#13255](matrix-org#13255), [\matrix-org#13276](matrix-org#13276)) - Support Implicit TLS (TLS without using a STARTTLS upgrade, typically on port 465) for sending emails, enabled by the new option `force_tls`. Contributed by Jan Schär. ([\matrix-org#13317](matrix-org#13317)) Bugfixes -------- - Fix a bug introduced in Synapse 1.15.0 where adding a user through the Synapse Admin API with a phone number would fail if the `enable_email_notifs` and `email_notifs_for_new_users` options were enabled. Contributed by @thomasweston12. ([\matrix-org#13263](matrix-org#13263)) - Fix a bug introduced in Synapse 1.40.0 where a user invited to a restricted room would be briefly unable to join. ([\matrix-org#13270](matrix-org#13270)) - Fix a long-standing bug where, in rare instances, Synapse could store the incorrect state for a room after a state resolution. ([\matrix-org#13278](matrix-org#13278)) - Fix a bug introduced in v1.18.0 where the `synapse_pushers` metric would overcount pushers when they are replaced. ([\matrix-org#13296](matrix-org#13296)) - Disable autocorrection and autocapitalisation on the username text field shown during registration when using SSO. ([\matrix-org#13350](matrix-org#13350)) - Update locked version of `frozendict` to 2.3.3, which has fixes for memory leaks affecting `/sync`. ([\matrix-org#13284](matrix-org#13284), [\matrix-org#13352](matrix-org#13352)) Improved Documentation ---------------------- - Provide an example of using the Admin API. Contributed by @jejo86. ([\matrix-org#13231](matrix-org#13231)) - Move the documentation for how URL previews work to the URL preview module. ([\matrix-org#13233](matrix-org#13233), [\matrix-org#13261](matrix-org#13261)) - Add another `contrib` script to help set up worker processes. Contributed by @villepeh. ([\matrix-org#13271](matrix-org#13271)) - Document that certain config options were added or changed in Synapse 1.62. Contributed by @behrmann. ([\matrix-org#13314](matrix-org#13314)) - Document the new `rc_invites.per_issuer` throttling option added in Synapse 1.63. ([\matrix-org#13333](matrix-org#13333)) - Mention that BuildKit is needed when building Docker images for tests. ([\matrix-org#13338](matrix-org#13338)) - Improve Caddy reverse proxy documentation. ([\matrix-org#13344](matrix-org#13344)) Deprecations and Removals ------------------------- - Drop tables that were formerly used for groups/communities. ([\matrix-org#12967](matrix-org#12967)) - Drop support for delegating email verification to an external server. ([\matrix-org#13192](matrix-org#13192)) - Drop support for calling `/_matrix/client/v3/account/3pid/bind` without an `id_access_token`, which was not permitted by the spec. Contributed by @Vetchu. ([\matrix-org#13239](matrix-org#13239)) - Stop building `.deb` packages for Ubuntu 21.10 (Impish Indri), which has reached end of life. ([\matrix-org#13326](matrix-org#13326)) Internal Changes ---------------- - Use lower transaction isolation level when purging rooms to avoid serialization errors. Contributed by Nick @ Beeper. ([\matrix-org#12942](matrix-org#12942)) - Remove code which incorrectly attempted to reconcile state with remote servers when processing incoming events. ([\matrix-org#12943](matrix-org#12943)) - Make the AS login method call `Auth.get_user_by_req` for checking the AS token. ([\matrix-org#13094](matrix-org#13094)) - Always use a version of canonicaljson that supports the C implementation of frozendict. ([\matrix-org#13172](matrix-org#13172)) - Add prometheus counters for ephemeral events and to device messages pushed to app services. Contributed by Brad @ Beeper. ([\matrix-org#13175](matrix-org#13175)) - Refactor receipts servlet logic to avoid duplicated code. ([\matrix-org#13198](matrix-org#13198)) - Preparation for database schema simplifications: populate `state_key` and `rejection_reason` for existing rows in the `events` table. ([\matrix-org#13215](matrix-org#13215)) - Remove unused database table `event_reference_hashes`. ([\matrix-org#13218](matrix-org#13218)) - Further reduce queries used sending events when creating new rooms. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13224](matrix-org#13224)) - Call the v2 identity service `/3pid/unbind` endpoint, rather than v1. Contributed by @Vetchu. ([\matrix-org#13240](matrix-org#13240)) - Use an asynchronous cache wrapper for the get event cache. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13242](matrix-org#13242), [\matrix-org#13308](matrix-org#13308)) - Optimise federation sender and appservice pusher event stream processing queries. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13251](matrix-org#13251)) - Log the stack when waiting for an entire room to be un-partial stated. ([\matrix-org#13257](matrix-org#13257)) - Fix spurious warning when fetching state after a missing prev event. ([\matrix-org#13258](matrix-org#13258)) - Clean-up tests for notifications. ([\matrix-org#13260](matrix-org#13260)) - Do not fail build if complement with workers fails. ([\matrix-org#13266](matrix-org#13266)) - Don't pull out state in `compute_event_context` for unconflicted state. ([\matrix-org#13267](matrix-org#13267), [\matrix-org#13274](matrix-org#13274)) - Reduce the rebuild time for the complement-synapse docker image. ([\matrix-org#13279](matrix-org#13279)) - Don't pull out the full state when creating an event. ([\matrix-org#13281](matrix-org#13281), [\matrix-org#13307](matrix-org#13307)) - Upgrade from Poetry 1.1.12 to 1.1.14, to fix bugs when locking packages. ([\matrix-org#13285](matrix-org#13285)) - Make `DictionaryCache` expire full entries if they haven't been queried in a while, even if specific keys have been queried recently. ([\matrix-org#13292](matrix-org#13292)) - Use `HTTPStatus` constants in place of literals in tests. ([\matrix-org#13297](matrix-org#13297)) - Improve performance of query `_get_subset_users_in_room_with_profiles`. ([\matrix-org#13299](matrix-org#13299)) - Up batch size of `bulk_get_push_rules` and `_get_joined_profiles_from_event_ids`. ([\matrix-org#13300](matrix-org#13300)) - Remove unnecessary `json.dumps` from tests. ([\matrix-org#13303](matrix-org#13303)) - Reduce memory usage of sending dummy events. ([\matrix-org#13310](matrix-org#13310)) - Prevent formatting changes of [matrix-org#3679](matrix-org#3679) from appearing in `git blame`. ([\matrix-org#13311](matrix-org#13311)) - Change `get_users_in_room` and `get_rooms_for_user` caches to enable pruning of old entries. ([\matrix-org#13313](matrix-org#13313)) - Validate federation destinations and log an error if a destination is invalid. ([\matrix-org#13318](matrix-org#13318)) - Fix `FederationClient.get_pdu()` returning events from the cache as `outliers` instead of original events we saw over federation. ([\matrix-org#13320](matrix-org#13320)) - Reduce memory usage of state caches. ([\matrix-org#13323](matrix-org#13323)) - Reduce the amount of state we store in the `state_cache`. ([\matrix-org#13324](matrix-org#13324)) - Add missing type hints to open tracing module. ([\matrix-org#13328](matrix-org#13328), [\matrix-org#13345](matrix-org#13345), [\matrix-org#13362](matrix-org#13362)) - Remove old base slaved store and de-duplicate cache ID generators. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13329](matrix-org#13329), [\matrix-org#13349](matrix-org#13349)) - When reporting metrics is enabled, use ~8x less data to describe DB transaction metrics. ([\matrix-org#13342](matrix-org#13342)) - Faster room joins: skip soft fail checks while Synapse only has partial room state, since the current membership of event senders may not be accurately known. ([\matrix-org#13354](matrix-org#13354)) # -----BEGIN PGP SIGNATURE----- # # iQIzBAABCgAdFiEE8SRSDO7gYkSP4chELS76LzL74EcFAmLo+zIACgkQLS76LzL7 # 4EehbRAAronXZtWM+ViMxPsiDj70KXYOKK117pGXK5XGf3Tyqb/vExA7c7bfimyW # d3FW855fe27AMsSfcMGDpxhggVa8sZDSdvQumt5jqDXrzC348mW/FYtgcYOxkoIa # Hh2/7V26CxWFsv8eVF3hwpualelT9lp2sedWXCQtdAkcQoWs2JwBsnoxSDliDZHg # jc4mBFBAkah5CJ3bcZuZXRsr9doKxDOAXUv19RXhdwEGO82mpSbwQ8P0mcw2S8zr # aAVza7jkVAza6ahg9qE0lMpi8uYE9/mt5JBnfrv/JxC7ZZfBg9jyHKaxFrzpjFsj # 3g0jhqzcNxRskD1sk1GKGVy7D9oTg1WVpii5l3M93KguSDLKxomouhgekWOxMPBe # 43xVdDI13ohsex+1QBnGnTSP7jZcfODnfvzSdyHQv6ef4k+OplRdfMA0QjkUcI5j # ocJlkm2D02vw1mnU3hHNdw9ri3vkaS1Qwfsz3ZEYgn6OcZOeKAWn351WMXF/F1fm # HYeQ5uMud+i+EekBtR8Op9ZICHt9Ogp49172enlSGzeyeD3yUk5HMAMrzJfmsp3W # /LCCONkRrV+R8TRByUQE9YtqxUgn+eSgB5Ew/2C/WB54pZHtco+rPqkY1Bhan4QJ # LeZTuzDKeXzgho1D5b4quEC2AWAqz3GeIvEVuOZCt8rJoMMRslg= # =RRRX # -----END PGP SIGNATURE----- # gpg: Signature made Tue Aug 2 11:23:46 2022 BST # gpg: using RSA key F124520CEEE062448FE1C8442D2EFA2F32FBE047 # gpg: Can't check signature: No public key # Conflicts: # synapse/rest/client/read_marker.py # synapse/rest/client/receipts.py # synapse/storage/databases/main/events_worker.py # synapse/storage/databases/main/purge_events.py # tests/rest/client/test_rooms.py # tests/storage/test_event_push_actions.py

erikjohnston added 12 commits July 15, 2022 15:26

Make DictionaryCache have better expiry properties

326a175

Newsfile

40a8fba

Fix literal

a22716c

Fix test

f046366

don't update access

602a81f

Fix mypy

23c2f39

Better stuff

cad555f

Fix up

7aceec3

Comments

057ae8b

Comment LruCache

7f7b36d

Comment LruCache

462db2a

Comment TreeCache

129691f

erikjohnston marked this pull request as ready for review July 18, 2022 11:59

erikjohnston requested a review from a team as a code owner July 18, 2022 11:59

erikjohnston commented Jul 18, 2022

View reviewed changes

tests/storage/test_state.py Show resolved Hide resolved

Merge branch 'develop' into erikj/dict_cache

45d0dce

richvdh self-assigned this Jul 19, 2022

richvdh suggested changes Jul 19, 2022

View reviewed changes

richvdh removed their assignment Jul 19, 2022

erikjohnston and others added 7 commits July 19, 2022 13:09

Update synapse/util/caches/dictionary_cache.py

ca8e1af

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

Update synapse/util/caches/lrucache.py

e4723df

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

Split out code to separate method

a74baa6

Mark DictionaryEntry as frozen

a9ebcd2

Don't reuse vars

6a76dba

Add example

d4133b2

Make LruCacheget_multi return something sane.

88aa56c

erikjohnston force-pushed the erikj/dict_cache branch from 2f8230e to 88aa56c Compare July 19, 2022 13:00

erikjohnston requested a review from richvdh July 19, 2022 13:18

richvdh suggested changes Jul 20, 2022

View reviewed changes

erikjohnston added 10 commits July 20, 2022 18:49

Make missing a list

12e14f2

Don't iterate twice over dict_key

fed7755

Clarify comment

c9f13f3

Use iterable rather than generator

67bb06d

Add example

2ec4cab

Add doc to TreeCache.get

50bb901

Note that if full is True known_absent must be empty

aa203c6

When fetching full dict don't return partial

2151474

Fix test, set takes an iterable

43e0030

Add simple test for invalidation

3c23161

erikjohnston force-pushed the erikj/dict_cache branch from 9d5d72f to 3c23161 Compare July 20, 2022 17:52

Fix test now we don't return partial dicts

e5ef14d

erikjohnston requested a review from richvdh July 21, 2022 08:33

richvdh reviewed Jul 21, 2022

View reviewed changes

synapse/util/caches/treecache.py Outdated Show resolved Hide resolved

tests/util/test_dict_cache.py Show resolved Hide resolved

tests/util/test_dict_cache.py Show resolved Hide resolved

erikjohnston and others added 3 commits July 21, 2022 13:51

Woo comments

10fb0d0

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

Comment if dict_keys is None

d521be2

Document update doesn't invalidate

3c84a98

erikjohnston requested a review from richvdh July 21, 2022 12:56

richvdh reviewed Jul 21, 2022

View reviewed changes

erikjohnston and others added 2 commits July 21, 2022 15:05

Update synapse/util/caches/dictionary_cache.py

9c54881

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

Flesh out return value

daa2741

erikjohnston requested a review from richvdh July 21, 2022 14:06

Lint

055a5dd

richvdh approved these changes Jul 21, 2022

View reviewed changes

erikjohnston merged commit 0b87eb8 into develop Jul 21, 2022

erikjohnston deleted the erikj/dict_cache branch July 21, 2022 16:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make DictionaryCache have better expiry properties #13292

Make DictionaryCache have better expiry properties #13292

erikjohnston commented Jul 15, 2022 •

edited

Loading

richvdh left a comment

erikjohnston commented Jul 19, 2022

richvdh left a comment

richvdh Jul 20, 2022

erikjohnston Jul 20, 2022

richvdh Jul 21, 2022

erikjohnston Jul 21, 2022

richvdh Jul 20, 2022

richvdh Jul 21, 2022

richvdh Jul 21, 2022

erikjohnston Jul 21, 2022

richvdh Jul 21, 2022

erikjohnston Jul 21, 2022

richvdh Jul 21, 2022

richvdh Jul 21, 2022

richvdh left a comment

richvdh left a comment

richvdh Jul 21, 2022

richvdh left a comment

erikjohnston commented Jul 21, 2022

Make DictionaryCache have better expiry properties #13292

Make DictionaryCache have better expiry properties #13292

Conversation

erikjohnston commented Jul 15, 2022 • edited Loading

richvdh left a comment

Choose a reason for hiding this comment

erikjohnston commented Jul 19, 2022

richvdh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

richvdh left a comment

Choose a reason for hiding this comment

richvdh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

richvdh left a comment

Choose a reason for hiding this comment

erikjohnston commented Jul 21, 2022

erikjohnston commented Jul 15, 2022 •

edited

Loading