Merge upstream 7-29-24 #17
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Re-introduces #17191, and includes #17197 and #17214 The basic idea is to stop calling `get_rooms_for_user` everywhere, and instead use the table `device_lists_changes_in_room`. Commits reviewable one-by-one.
This PR ports the logic from the [synapse_auto_accept_invite](https://github.com/matrix-org/synapse-auto-accept-invite) module into synapse. I went with the naive approach of injecting the "module" next to where third party modules are currently loaded. If there is a better/preferred way to handle this, I'm all ears. It wasn't obvious to me if there was a better location to add this logic that would cleanly apply to all incoming invite events. Relies on element-hq/synapse#17166 to fix linter errors.
Update OIDC documentation: by default Matrix doesn't query userinfo endpoint, then claims should be put on id_token.
…atabase usage. (#17219) The log format is the same as the request log format, except: - fields that are specific to HTTP requests have been removed - the task's params are included at the end of the log line. These log lines are emitted: - when the task function finishes — both completion and failure (and I suppose it is possible for a task to become schedulable again?) - every 5 minutes whilst it is running Closes #17217. --------- Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
…uto_join_rooms` option. (#17176) Would have been useful for tracking down #16878. Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
This is being introduced as part of Sliding Sync but doesn't have any sliding window component. It's just a way to get E2EE events without having to sit through a big initial sync (`/sync` v2). And we can avoid encryption events being backed up by the main sync response or vice-versa. Part of some Sliding Sync simplification/experimentation. See [this discussion](element-hq/synapse#17167 (comment)) for why it may not be as useful as we thought. Based on: - matrix-org/matrix-spec-proposals#3575 - matrix-org/matrix-spec-proposals#3885 - matrix-org/matrix-spec-proposals#3884
…` (#17213) [MSC3916](https://github.com/matrix-org/matrix-spec-proposals/blob/rav/authentication-for-media/proposals/3916-authentication-for-media.md) adds new media endpoints under `_matrix/client`. This PR adds the `/preview_url`, `/config`, and `/thumbnail` endpoints. `/download` will be added in a follow-up PR once the work for the federation `/download` endpoint is complete (see element-hq/synapse#17172). Should be reviewable commit-by-commit.
When a module rejects a piece of media we end up trying to close the same logging context twice. Instead of fixing the existing code we refactor to use an async context manager, which is easier to write correctly.
Currently this causes one of then to 500.
Currently sending a to-device message to a user ID with a dodgy destination is accepted, but then ends up spamming the logs when we try and send to the destination. An alternative would be to reject the request, but I'm slightly nervous that could break things.
We relax this as there are use cases where this is safe, though it is still highly recommended that people avoid using it.
This is a tree cache already, so may as well move the room ID to the front and use that
There is a problem with `StreamIdGenerator` where it can go backwards over restarts when a stream ID is requested but then not inserted into the DB. This is problematic if we want to land #17215, and is generally a potential cause for all sorts of nastiness. Instead of trying to fix `StreamIdGenerator`, we may as well move to `MultiWriterIdGenerator` that does not suffer from this problem (the latest positions are stored in `stream_positions` table). This involves adding SQLite support to the class. This only changes id generators that were already using `MultiWriterIdGenerator` under postgres, a separate PR will move the rest of the uses of `StreamIdGenerator` over.
Fixes up #17239 We need to keep the spam check within the `try/except` block. Also makes it so that we don't enter the top span twice. Also also ensures that we get the right thumbnail length.
We started ensuring we only insert valid destinations: element-hq/synapse#17240
…or` (#17229) Replaces all usages of `StreamIdGenerator` with `MultiWriterIdGenerator`, which is safer.
…ups. (#17164) We try and deduplicate in two places: 1) really early on, and 2) just before we persist the event. The first case was broken due to it occuring before the profile information was added, and so it thought the event contents were different. The second case did catch it and handle it correctly, however doing so creates a redundant state group leading to bloat. Fixes #3791
Otherwise things will get confused. An alternative would be to make sure that for lagging stream we don't return anything (and make sure the returned next_batch token doesn't go backwards). But that is a faff.
This was broken by the sentry 2.0 upgrade Broke in v1.108.0
Bumps [hiredis](https://github.com/redis/hiredis-py) from 2.3.2 to 3.0.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/redis/hiredis-py/releases">hiredis's releases</a>.</em></p> <blockquote> <h2>3.0.0</h2> <h1>Changes</h1> <h2>Breaking Changes</h2> <ul> <li>Return Redis sets as Python lists (<a href="https://redirect.github.com/redis/hiredis-py/issues/189">#189</a>)</li> </ul> <h2>🐛 Bug Fixes</h2> <ul> <li>Return Redis sets as Python lists (<a href="https://redirect.github.com/redis/hiredis-py/issues/189">#189</a>)</li> </ul> <h2>Contributors</h2> <p>We'd like to thank all the contributors who worked on this release!</p> <p><a href="https://github.com/gerzse"><code>@gerzse</code></a></p> <h2>2.4.0</h2> <h1>Changes</h1> <h2>🧰 Maintenance</h2> <ul> <li>Fix small typo (<a href="https://redirect.github.com/redis/hiredis-py/issues/192">#192</a>)</li> <li>Quote version for Python setup action in CI (<a href="https://redirect.github.com/redis/hiredis-py/issues/191">#191</a>)</li> <li>Fix building the wheel for windows (<a href="https://redirect.github.com/redis/hiredis-py/issues/190">#190</a>)</li> <li>pack: Replace sdsalloc.h with alloc.h (<a href="https://redirect.github.com/redis/hiredis-py/issues/159">#159</a>)</li> <li>Bump black from 22.3.0 to 24.3.0 (<a href="https://redirect.github.com/redis/hiredis-py/issues/185">#185</a>)</li> <li>Removing python 3.7 trove (<a href="https://redirect.github.com/redis/hiredis-py/issues/181">#181</a>)</li> <li>Badge for latest released on Pypi (<a href="https://redirect.github.com/redis/hiredis-py/issues/182">#182</a>)</li> <li>Sync license in metadata with LICENSE file (<a href="https://redirect.github.com/redis/hiredis-py/issues/183">#183</a>)</li> </ul> <h2>Contributors</h2> <p>We'd like to thank all the contributors who worked on this release!</p> <p><a href="https://github.com/Apteryks"><code>@Apteryks</code></a>, <a href="https://github.com/ArtemIsmagilov"><code>@ArtemIsmagilov</code></a>, <a href="https://github.com/chayim"><code>@chayim</code></a>, <a href="https://github.com/dependabot"><code>@dependabot</code></a>, <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot], <a href="https://github.com/gerzse"><code>@gerzse</code></a> and <a href="https://github.com/shadchin"><code>@shadchin</code></a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/redis/hiredis-py/commit/c1eefbdb76614435f7433207bf385ba8cb930b60"><code>c1eefbd</code></a> Return Redis sets as Python lists (<a href="https://redirect.github.com/redis/hiredis-py/issues/189">#189</a>)</li> <li><a href="https://github.com/redis/hiredis-py/commit/a94bb447173c60b90709a9ed117b3a5b699ff795"><code>a94bb44</code></a> Version 2.4.0 (<a href="https://redirect.github.com/redis/hiredis-py/issues/193">#193</a>)</li> <li><a href="https://github.com/redis/hiredis-py/commit/7792dd23384aa7b00761df51d8011fe24fa33143"><code>7792dd2</code></a> Fix a typo in the README file (<a href="https://redirect.github.com/redis/hiredis-py/issues/192">#192</a>)</li> <li><a href="https://github.com/redis/hiredis-py/commit/01fa2fd6f123e3424ffa00a647f2bf83d48543be"><code>01fa2fd</code></a> Quote version for Python setup action in CI (<a href="https://redirect.github.com/redis/hiredis-py/issues/191">#191</a>)</li> <li><a href="https://github.com/redis/hiredis-py/commit/4c970a336567223573c700a44e405a0c263a85fa"><code>4c970a3</code></a> Fix building the wheel for windows (<a href="https://redirect.github.com/redis/hiredis-py/issues/190">#190</a>)</li> <li><a href="https://github.com/redis/hiredis-py/commit/f4dd0814c16dc9a8efa72434101d49c97778c830"><code>f4dd081</code></a> pack: Replace sdsalloc.h with alloc.h (<a href="https://redirect.github.com/redis/hiredis-py/issues/159">#159</a>)</li> <li><a href="https://github.com/redis/hiredis-py/commit/e70af5b94f0881f23f2fb35417bb2e0d0b792f53"><code>e70af5b</code></a> Bump black from 22.3.0 to 24.3.0 (<a href="https://redirect.github.com/redis/hiredis-py/issues/185">#185</a>)</li> <li><a href="https://github.com/redis/hiredis-py/commit/cc239705fb64f92c6ac3aff36679a300662e0ca7"><code>cc23970</code></a> Removing Python 3.7 trove (<a href="https://redirect.github.com/redis/hiredis-py/issues/181">#181</a>)</li> <li><a href="https://github.com/redis/hiredis-py/commit/64e3394548fe670e7853a2407799e13daa4bf2cb"><code>64e3394</code></a> Badge for latest released on Pypi (<a href="https://redirect.github.com/redis/hiredis-py/issues/182">#182</a>)</li> <li><a href="https://github.com/redis/hiredis-py/commit/ba18089d74f991b9b9e7ac9c17f85432f50d0048"><code>ba18089</code></a> Sync license in metadata with LICENSE file (<a href="https://redirect.github.com/redis/hiredis-py/issues/183">#183</a>)</li> <li>See full diff in <a href="https://github.com/redis/hiredis-py/compare/v2.3.2...v3.0.0">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…17231) This is to address an issue in which `m.presence` results on initial sync are not returning entries of users who are currently offline. The original behaviour was from element-hq/synapse#1535 This change is useful for applications that use the presence system for tracking user profile information/updates (e.g. element-hq/synapse#16992 or for profile status messages). This is gated behind a new configuration option to avoid performance impact for applications that don't need this, as a pragmatic solution for now.
…ced in v1.109.0. (#17428) Introduced in: #17215 This caused us a minor bit of grief as the volume of logs produced was much higher than normal --------- Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
We do this by bulk fetching the latest stream ordering. --------- Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Update debian template - new link to the delegation docs
This is in preparation for adding per-connection state. --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
Backfill events have a negative stream ordering, and so its not useful to use to compare with other (positive) stream orderings. Plus, the Rust SDK currently assumes `bump_stamp` is positive.
Extensions based on [MSC3575](matrix-org/matrix-spec-proposals#3575): Sliding Sync
…al_3pids (#17476) Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
…pt. 1) (#17481) `SlidingSyncBase` for tests was first introduced in element-hq/synapse#17452 Part 2: element-hq/synapse#17482
…_sync(...)` (pt. 2) (#17482) `SlidingSyncBase.do_sync()` for tests was first introduced in element-hq/synapse#17452 Part 1: element-hq/synapse#17481
…gs for selective presence tracking Signed-off-by: Michael Hollister <michael@futo.org>
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
…gs for selective presence tracking Additionally switched schema versions 86 with 87 since presence updates were applied eariler Signed-off-by: Michael Hollister <michael@futo.org>
Weakness in auth chain indexing allows DoS from remote room members through disk fill and high CPU usage. A remote Matrix user with malicious intent, sharing a room with Synapse instances before 1.104.1, can dispatch specially crafted events to exploit a weakness in how the auth chain cover index is calculated. This can induce high CPU consumption and accumulate excessive data in the database of such instances, resulting in a denial of service. Servers in private federations, or those that do not federate, are not affected.
…gs for selective presence tracking Signed-off-by: Michael Hollister <michael@futo.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Additional changes:
michael/presence-initial-syncchanges are now deprecated since upstream Synapse has merged the PR Fixed presence results not returning offline users on initial sync element-hq/synapse#1723186and87have been flipped for upgrade compatibility since upstream PR is still pending merge: Added presence update on change of profile information and config flags for selective presence tracking element-hq/synapse#16992