Merge tag 'v1.80.0' into merge-1.78

Synapse 1.80.0 (2023-03-28)
===========================

No significant changes since 1.80.0rc2.

Synapse 1.80.0rc2 (2023-03-22)
==============================

Bugfixes
--------

- Fix a bug in which the [`POST /_matrix/client/v3/rooms/{roomId}/report/{eventId}`](https://spec.matrix.org/v1.6/client-server-api/#post_matrixclientv3roomsroomidreporteventid) endpoint would return the wrong error if the user did not have permission to view the event. This aligns Synapse's implementation with [MSC2249](matrix-org/matrix-spec-proposals#2249). ([\matrix-org#15298](matrix-org#15298), [\matrix-org#15300](matrix-org#15300))
- Fix a bug introduced in Synapse 1.75.0rc1 where the [SQLite port_db script](https://matrix-org.github.io/synapse/latest/postgres.html#porting-from-sqlite)
  would fail to open the SQLite database. ([\matrix-org#15301](matrix-org#15301))

Synapse 1.80.0rc1 (2023-03-21)
==============================

Features
--------

- Stabilise support for [MSC3966](matrix-org/matrix-spec-proposals#3966): `event_property_contains` push condition. ([\matrix-org#15187](matrix-org#15187))
- Implement [MSC2659](matrix-org/matrix-spec-proposals#2659): application service ping endpoint. Contributed by Tulir @ Beeper. ([\matrix-org#15249](matrix-org#15249))
- Allow loading `/register/available` endpoint on workers. ([\matrix-org#15268](matrix-org#15268))
- Improve performance of creating and authenticating events. ([\matrix-org#15195](matrix-org#15195))
- Add topic and name events to group of events that are batch persisted when creating a room. ([\matrix-org#15229](matrix-org#15229))

Bugfixes
--------

- Fix a long-standing bug in which the user directory would assume any remote membership state events represent a profile change. ([\matrix-org#14755](matrix-org#14755), [\matrix-org#14756](matrix-org#14756))
- Implement [MSC3873](matrix-org/matrix-spec-proposals#3873) to fix a long-standing bug where properties with dots were handled ambiguously in push rules. ([\matrix-org#15190](matrix-org#15190))
- Faster joins: Fix a bug introduced in Synapse 1.66 where spurious "Failed to find memberships ..." errors would be logged. ([\matrix-org#15232](matrix-org#15232))
- Fix a long-standing error when sending message into deleted room. ([\matrix-org#15235](matrix-org#15235))

Updates to the Docker image
---------------------------

- Ensure the Dockerfile builds on platforms that don't have a `cryptography` wheel. ([\matrix-org#15239](matrix-org#15239))
- Mirror images to the GitHub Container Registry (`ghcr.io/matrix-org/synapse`). ([\matrix-org#15281](matrix-org#15281), [\matrix-org#15282](matrix-org#15282))

Improved Documentation
----------------------

- Add a missing endpoint to the workers documentation. ([\matrix-org#15223](matrix-org#15223))

Internal Changes
----------------

- Add additional functionality to declaring worker types when starting Complement in worker mode. ([\matrix-org#14921](matrix-org#14921))
- Add `Synapse-Trace-Id` to `access-control-expose-headers` header. ([\matrix-org#14974](matrix-org#14974))
- Make the `HttpTransactionCache` use the `Requester` in addition of the just the `Request` to build the transaction key. ([\matrix-org#15200](matrix-org#15200))
- Improve log lines when purging rooms. ([\matrix-org#15222](matrix-org#15222))
- Improve type hints. ([\matrix-org#15230](matrix-org#15230), [\matrix-org#15231](matrix-org#15231), [\matrix-org#15238](matrix-org#15238))
- Move various module API callback registration methods to a dedicated class. ([\matrix-org#15237](matrix-org#15237))
- Configure GitHub Actions for merge queues. ([\matrix-org#15244](matrix-org#15244))
- Add schema comments about the `destinations` and `destination_rooms` tables. ([\matrix-org#15247](matrix-org#15247))
- Skip processing of auto-join room behaviour if there are no auto-join rooms configured. ([\matrix-org#15262](matrix-org#15262))
- Remove unused store method `_set_destination_retry_timings_emulated`. ([\matrix-org#15266](matrix-org#15266))
- Reorganize URL preview code. ([\matrix-org#15269](matrix-org#15269))
- Clean-up direct TCP replication code. ([\matrix-org#15272](matrix-org#15272), [\matrix-org#15274](matrix-org#15274))
- Make `configure_workers_and_start` script used in Complement tests compatible with older versions of Python. ([\matrix-org#15275](matrix-org#15275))
- Add a `/versions` flag for [MSC3952](matrix-org/matrix-spec-proposals#3952). ([\matrix-org#15293](matrix-org#15293))
- Bump hiredis from 2.2.1 to 2.2.2. ([\matrix-org#15252](matrix-org#15252))
- Bump serde from 1.0.152 to 1.0.155. ([\matrix-org#15253](matrix-org#15253))
- Bump pysaml2 from 7.2.1 to 7.3.1. ([\matrix-org#15254](matrix-org#15254))
- Bump msgpack from 1.0.4 to 1.0.5. ([\matrix-org#15255](matrix-org#15255))
- Bump gitpython from 3.1.30 to 3.1.31. ([\matrix-org#15256](matrix-org#15256))
- Bump cryptography from 39.0.1 to 39.0.2. ([\matrix-org#15257](matrix-org#15257))
- Bump pydantic from 1.10.4 to 1.10.6. ([\matrix-org#15286](matrix-org#15286))
- Bump serde from 1.0.155 to 1.0.157. ([\matrix-org#15287](matrix-org#15287))
- Bump anyhow from 1.0.69 to 1.0.70. ([\matrix-org#15288](matrix-org#15288))
- Bump txredisapi from 1.4.7 to 1.4.9. ([\matrix-org#15289](matrix-org#15289))
- Bump pygithub from 1.57 to 1.58.1. ([\matrix-org#15290](matrix-org#15290))
- Bump types-requests from 2.28.11.12 to 2.28.11.15. ([\matrix-org#15291](matrix-org#15291))

.ci/scripts/calculate_jobs.py

@@ -109,11 +109,26 @@ def set_output(key: str, value: str):
  "postgres": "multi-postgres",
  "workers": "workers",
  },
+ {
+ "sytest-tag": "focal",
+ "postgres": "multi-postgres",
+ "workers": "workers",
+ "reactor": "asyncio",
+ },
 ]
 
 if not IS_PR:
  sytest_tests.extend(
  [
+ {
+ "sytest-tag": "focal",
+ "reactor": "asyncio",
+ },
+ {
+ "sytest-tag": "focal",
+ "postgres": "postgres",
+ "reactor": "asyncio",
+ },
  {
  "sytest-tag": "testing",
  "postgres": "postgres",

.git-blame-ignore-revs

@@ -21,4 +21,8 @@ aff1eb7c671b0a3813407321d2702ec46c71fa56
 0a00b7ff14890987f09112a2ae696c61001e6cf1
 
 # Convert tests/rest/admin/test_room.py to unix file endings (#7953).
-c4268e3da64f1abb5b31deaeb5769adb6510c0a7
+c4268e3da64f1abb5b31deaeb5769adb6510c0a7
+
+# Update black to 23.1.0 (#15103)
+9bb2eac71962970d02842bca441f4bcdbbf93a11
+

.github/workflows/docker.yml

@@ -10,6 +10,7 @@ on:
 
 permissions:
  contents: read
+ packages: write
 
 jobs:
  build:
@@ -34,11 +35,20 @@ jobs:
  username: ${{ secrets.DOCKERHUB_USERNAME }}
  password: ${{ secrets.DOCKERHUB_TOKEN }}
 
+ - name: Log in to GHCR
+ uses: docker/login-action@v2
+ with:
+ registry: ghcr.io
+ username: ${{ github.repository_owner }}
+ password: ${{ secrets.GITHUB_TOKEN }}
+
  - name: Calculate docker image tag
  id: set-tag
  uses: docker/metadata-action@master
  with:
- images: matrixdotorg/synapse
+ images: |
+ docker.io/matrixdotorg/synapse
+ ghcr.io/matrix-org/synapse
  flavor: |
  latest=false
  tags: |

.github/workflows/docs-pr-netlify.yaml

@@ -14,7 +14,7 @@ jobs:
  # There's a 'download artifact' action, but it hasn't been updated for the workflow_run action
  # (https://github.com/actions/download-artifact/issues/60) so instead we get this mess:
  - name: 📥 Download artifact
- uses: dawidd6/action-download-artifact@bd10f381a96414ce2b13a11bfa89902ba7cea07f # v2.24.3
+ uses: dawidd6/action-download-artifact@5e780fc7bbd0cac69fc73271ed86edf5dcb72d67 # v2.26.0
  with:
  workflow: docs-pr.yaml
  run_id: ${{ github.event.workflow_run.id }}

.github/workflows/docs-pr.yaml

@@ -12,7 +12,7 @@ jobs:
  name: GitHub Pages
  runs-on: ubuntu-latest
  steps:
- - uses: actions/checkout@v2
+ - uses: actions/checkout@v3
 
  - name: Setup mdbook
  uses: peaceiris/actions-mdbook@adeb05db28a0c0004681db83893d56c0388ea9ea # v1.2.0
@@ -39,7 +39,7 @@ jobs:
  name: Check links in documentation
  runs-on: ubuntu-latest
  steps:
- - uses: actions/checkout@v2
+ - uses: actions/checkout@v3
 
  - name: Setup mdbook
  uses: peaceiris/actions-mdbook@adeb05db28a0c0004681db83893d56c0388ea9ea # v1.2.0

.github/workflows/latest_deps.yml

@@ -27,7 +27,7 @@ jobs:
  steps:
  - uses: actions/checkout@v3
  - name: Install Rust
- uses: dtolnay/rust-toolchain@9cd00a88a73addc8617065438eff914dd08d0955
+ uses: dtolnay/rust-toolchain@e12eda571dc9a5ee5d58eecf4738ec291c66f295
  with:
  toolchain: stable
  - uses: Swatinem/rust-cache@v2
@@ -61,7 +61,7 @@ jobs:
  - uses: actions/checkout@v3
 
  - name: Install Rust
- uses: dtolnay/rust-toolchain@9cd00a88a73addc8617065438eff914dd08d0955
+ uses: dtolnay/rust-toolchain@e12eda571dc9a5ee5d58eecf4738ec291c66f295
  with:
  toolchain: stable
  - uses: Swatinem/rust-cache@v2
@@ -134,7 +134,7 @@ jobs:
  - uses: actions/checkout@v3
 
  - name: Install Rust
- uses: dtolnay/rust-toolchain@9cd00a88a73addc8617065438eff914dd08d0955
+ uses: dtolnay/rust-toolchain@e12eda571dc9a5ee5d58eecf4738ec291c66f295
  with:
  toolchain: stable
  - uses: Swatinem/rust-cache@v2

.github/workflows/poetry_lockfile.yaml

@@ -0,0 +1,24 @@
+on:
+ push:
+ branches: ["develop", "release-*"]
+ paths:
+ - poetry.lock
+ pull_request:
+ paths:
+ - poetry.lock
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.ref }}
+ cancel-in-progress: true
+
+jobs:
+ check-sdists:
+ name: "Check locked dependencies have sdists"
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v3
+ - uses: actions/setup-python@v4
+ with:
+ python-version: '3.x'
+ - run: pip install tomli
+ - run: ./scripts-dev/check_locked_deps_have_sdists.py

.github/workflows/push_complement_image.yml

@@ -48,7 +48,7 @@ jobs:
  with:
  ref: master
  - name: Login to registry
- uses: docker/login-action@v1
+ uses: docker/login-action@v2
  with:
  registry: ghcr.io
  username: ${{ github.actor }}

.github/workflows/release-artifacts.yml

@@ -4,13 +4,15 @@ name: Build release artifacts
 
 on:
  # we build on PRs and develop to (hopefully) get early warning
- # of things breaking (but only build one set of debs)
+ # of things breaking (but only build one set of debs). PRs skip
+ # building wheels on macOS & ARM.
  pull_request:
  push:
  branches: ["develop", "release-*"]
 
  # we do the full build on tags.
  tags: ["v*"]
+ merge_group:
  workflow_dispatch:
 
 concurrency:

.github/workflows/tests.yml

@@ -4,6 +4,7 @@ on:
  push:
  branches: ["develop", "release-*"]
  pull_request:
+ merge_group:
  workflow_dispatch:
 
 concurrency:
@@ -112,7 +113,7 @@ jobs:
  # There don't seem to be versioned releases of this action per se: for each rust
  # version there is a branch which gets constantly rebased on top of master.
  # We pin to a specific commit for paranoia's sake.
- uses: dtolnay/rust-toolchain@9cd00a88a73addc8617065438eff914dd08d0955
+ uses: dtolnay/rust-toolchain@e12eda571dc9a5ee5d58eecf4738ec291c66f295
  with:
  toolchain: 1.58.1
  components: clippy
@@ -134,7 +135,7 @@ jobs:
  # There don't seem to be versioned releases of this action per se: for each rust
  # version there is a branch which gets constantly rebased on top of master.
  # We pin to a specific commit for paranoia's sake.
- uses: dtolnay/rust-toolchain@9cd00a88a73addc8617065438eff914dd08d0955
+ uses: dtolnay/rust-toolchain@e12eda571dc9a5ee5d58eecf4738ec291c66f295
  with:
  toolchain: nightly-2022-12-01
  components: clippy
@@ -154,9 +155,10 @@ jobs:
  # There don't seem to be versioned releases of this action per se: for each rust
  # version there is a branch which gets constantly rebased on top of master.
  # We pin to a specific commit for paranoia's sake.
- uses: dtolnay/rust-toolchain@9cd00a88a73addc8617065438eff914dd08d0955
+ uses: dtolnay/rust-toolchain@e12eda571dc9a5ee5d58eecf4738ec291c66f295
  with:
- toolchain: 1.58.1
+ # We use nightly so that it correctly groups together imports
+ toolchain: nightly-2022-12-01
  components: rustfmt
  - uses: Swatinem/rust-cache@v2
 
@@ -221,7 +223,7 @@ jobs:
  # There don't seem to be versioned releases of this action per se: for each rust
  # version there is a branch which gets constantly rebased on top of master.
  # We pin to a specific commit for paranoia's sake.
- uses: dtolnay/rust-toolchain@9cd00a88a73addc8617065438eff914dd08d0955
+ uses: dtolnay/rust-toolchain@e12eda571dc9a5ee5d58eecf4738ec291c66f295
  with:
  toolchain: 1.58.1
  - uses: Swatinem/rust-cache@v2
@@ -266,7 +268,7 @@ jobs:
  # There don't seem to be versioned releases of this action per se: for each rust
  # version there is a branch which gets constantly rebased on top of master.
  # We pin to a specific commit for paranoia's sake.
- uses: dtolnay/rust-toolchain@9cd00a88a73addc8617065438eff914dd08d0955
+ uses: dtolnay/rust-toolchain@e12eda571dc9a5ee5d58eecf4738ec291c66f295
  with:
  toolchain: 1.58.1
  - uses: Swatinem/rust-cache@v2
@@ -368,6 +370,7 @@ jobs:
  SYTEST_BRANCH: ${{ github.head_ref }}
  POSTGRES: ${{ matrix.job.postgres && 1}}
  MULTI_POSTGRES: ${{ (matrix.job.postgres == 'multi-postgres') && 1}}
+ ASYNCIO_REACTOR: ${{ (matrix.job.reactor == 'asyncio') && 1 }}
  WORKERS: ${{ matrix.job.workers && 1 }}
  BLACKLIST: ${{ matrix.job.workers && 'synapse-blacklist-with-workers' }}
  TOP: ${{ github.workspace }}
@@ -386,7 +389,7 @@ jobs:
  # There don't seem to be versioned releases of this action per se: for each rust
  # version there is a branch which gets constantly rebased on top of master.
  # We pin to a specific commit for paranoia's sake.
- uses: dtolnay/rust-toolchain@9cd00a88a73addc8617065438eff914dd08d0955
+ uses: dtolnay/rust-toolchain@e12eda571dc9a5ee5d58eecf4738ec291c66f295
  with:
  toolchain: 1.58.1
  - uses: Swatinem/rust-cache@v2
@@ -531,7 +534,7 @@ jobs:
  # There don't seem to be versioned releases of this action per se: for each rust
  # version there is a branch which gets constantly rebased on top of master.
  # We pin to a specific commit for paranoia's sake.
- uses: dtolnay/rust-toolchain@9cd00a88a73addc8617065438eff914dd08d0955
+ uses: dtolnay/rust-toolchain@e12eda571dc9a5ee5d58eecf4738ec291c66f295
  with:
  toolchain: 1.58.1
  - uses: Swatinem/rust-cache@v2
@@ -562,7 +565,7 @@ jobs:
  # There don't seem to be versioned releases of this action per se: for each rust
  # version there is a branch which gets constantly rebased on top of master.
  # We pin to a specific commit for paranoia's sake.
- uses: dtolnay/rust-toolchain@9cd00a88a73addc8617065438eff914dd08d0955
+ uses: dtolnay/rust-toolchain@e12eda571dc9a5ee5d58eecf4738ec291c66f295
  with:
  toolchain: 1.58.1
  - uses: Swatinem/rust-cache@v2
@@ -585,7 +588,7 @@ jobs:
  # There don't seem to be versioned releases of this action per se: for each rust
  # version there is a branch which gets constantly rebased on top of master.
  # We pin to a specific commit for paranoia's sake.
- uses: dtolnay/rust-toolchain@9cd00a88a73addc8617065438eff914dd08d0955
+ uses: dtolnay/rust-toolchain@e12eda571dc9a5ee5d58eecf4738ec291c66f295
  with:
  toolchain: nightly-2022-12-01
  - uses: Swatinem/rust-cache@v2

.github/workflows/triage-incoming.yml

@@ -6,7 +6,7 @@ on:
 
 jobs:
  triage:
- uses: matrix-org/backend-meta/.github/workflows/triage-incoming.yml@v1
+ uses: matrix-org/backend-meta/.github/workflows/triage-incoming.yml@v2
  with: 
  project_id: 'PVT_kwDOAIB0Bs4AFDdZ'
  content_id: ${{ github.event.issue.node_id }}

.github/workflows/twisted_trunk.yml

@@ -18,7 +18,7 @@ jobs:
  - uses: actions/checkout@v3
 
  - name: Install Rust
- uses: dtolnay/rust-toolchain@9cd00a88a73addc8617065438eff914dd08d0955
+ uses: dtolnay/rust-toolchain@e12eda571dc9a5ee5d58eecf4738ec291c66f295
  with:
  toolchain: stable
  - uses: Swatinem/rust-cache@v2
@@ -43,7 +43,7 @@ jobs:
  - run: sudo apt-get -qq install xmlsec1
 
  - name: Install Rust
- uses: dtolnay/rust-toolchain@9cd00a88a73addc8617065438eff914dd08d0955
+ uses: dtolnay/rust-toolchain@e12eda571dc9a5ee5d58eecf4738ec291c66f295
  with:
  toolchain: stable
  - uses: Swatinem/rust-cache@v2
@@ -82,7 +82,7 @@ jobs:
  - uses: actions/checkout@v3
 
  - name: Install Rust
- uses: dtolnay/rust-toolchain@9cd00a88a73addc8617065438eff914dd08d0955
+ uses: dtolnay/rust-toolchain@e12eda571dc9a5ee5d58eecf4738ec291c66f295
  with:
  toolchain: stable
  - uses: Swatinem/rust-cache@v2