Merge tag 'v1.72.0' into merge-v1.72

Synapse 1.72.0 (2022-11-22)
===========================

Please note that Synapse now only supports PostgreSQL 11+, because PostgreSQL 10 has reached end-of-life, c.f. our [Deprecation Policy](https://github.com/matrix-org/synapse/blob/develop/docs/deprecation_policy.md).

Bugfixes
--------

- Update forgotten references to legacy metrics in the included Grafana dashboard. ([\matrix-org#14477](matrix-org#14477))

Synapse 1.72.0rc1 (2022-11-16)
==============================

Features
--------

- Add experimental support for [MSC3912](matrix-org/matrix-spec-proposals#3912): Relation-based redactions. ([\matrix-org#14260](matrix-org#14260))
- Build Debian packages for Ubuntu 22.10 (Kinetic Kudu). ([\matrix-org#14396](matrix-org#14396))
- Add an [Admin API](https://matrix-org.github.io/synapse/latest/usage/administration/admin_api/index.html) endpoint for user lookup based on third-party ID (3PID). Contributed by @ashfame. ([\matrix-org#14405](matrix-org#14405))
- Faster joins: include heroes' membership events in the partial join response, for rooms without a name or canonical alias. ([\matrix-org#14442](matrix-org#14442))

Bugfixes
--------

- Faster joins: do not block creation of or queries for room aliases during the resync. ([\matrix-org#14292](matrix-org#14292))
- Fix a bug introduced in Synapse 1.64.0rc1 which could cause log spam when fetching events from other homeservers. ([\matrix-org#14347](matrix-org#14347))
- Fix a bug introduced in 1.66 which would not send certain pushrules to clients. Contributed by Nico. ([\matrix-org#14356](matrix-org#14356))
- Fix a bug introduced in v1.71.0rc1 where the power level event was incorrectly created during initial room creation. ([\matrix-org#14361](matrix-org#14361))
- Fix the refresh token endpoint to be under /r0 and /v3 instead of /v1. Contributed by Tulir @ Beeper. ([\matrix-org#14364](matrix-org#14364))
- Fix a long-standing bug where Synapse would raise an error when encountering an unrecognised field in a `/sync` filter, instead of ignoring it for forward compatibility. ([\matrix-org#14369](matrix-org#14369))
- Fix a background database update, introduced in Synapse 1.64.0, which could cause poor database performance. ([\matrix-org#14374](matrix-org#14374))
- Fix PostgreSQL sometimes using table scans for queries against the `event_search` table, taking a long time and a large amount of IO. ([\matrix-org#14409](matrix-org#14409))
- Fix rendering of some HTML templates (including emails). Introduced in v1.71.0. ([\matrix-org#14448](matrix-org#14448))
- Fix a bug introduced in Synapse 1.70.0 where the background updates to add non-thread unique indexes on receipts could fail when upgrading from 1.67.0 or earlier. ([\matrix-org#14453](matrix-org#14453))

Updates to the Docker image
---------------------------

- Add all Stream Writer worker types to `configure_workers_and_start.py`. ([\matrix-org#14197](matrix-org#14197))
- Remove references to legacy worker types in the multi-worker Dockerfile. ([\matrix-org#14294](matrix-org#14294))

Improved Documentation
----------------------

- Upload documentation PRs to Netlify. ([\matrix-org#12947](matrix-org#12947), [\matrix-org#14370](matrix-org#14370))
- Add addtional TURN server configuration example based on [eturnal](https://github.com/processone/eturnal) and adjust general TURN server doc structure. ([\matrix-org#14293](matrix-org#14293))
- Add example on how to load balance /sync requests. Contributed by [aceArt](https://aceart.de). ([\matrix-org#14297](matrix-org#14297))
- Edit sample Nginx reverse proxy configuration to use HTTP/1.1. Contributed by Brad Jones. ([\matrix-org#14414](matrix-org#14414))

Deprecations and Removals
-------------------------

- Remove support for PostgreSQL 10. ([\matrix-org#14392](matrix-org#14392), [\matrix-org#14397](matrix-org#14397))

Internal Changes
----------------

- Run unit tests against Python 3.11. ([\matrix-org#13812](matrix-org#13812))
- Add TLS support for generic worker endpoints. ([\matrix-org#14128](matrix-org#14128), [\matrix-org#14455](matrix-org#14455))
- Switch to a maintained action for installing Rust in CI. ([\matrix-org#14313](matrix-org#14313))
- Add override ability to `complement.sh` command line script to request certain types of workers. ([\matrix-org#14324](matrix-org#14324))
- Enabling testing of [MSC3874](matrix-org/matrix-spec-proposals#3874) (filtering of `/messages` by relation type) in complement. ([\matrix-org#14339](matrix-org#14339))
- Concisely log a failure to resolve state due to missing `prev_events`. ([\matrix-org#14346](matrix-org#14346))
- Use a maintained Github action to install Rust. ([\matrix-org#14351](matrix-org#14351))
- Cleanup old worker datastore classes. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#14375](matrix-org#14375))
- Test against PostgreSQL 15 in CI. ([\matrix-org#14394](matrix-org#14394))
- Remove unreachable code. ([\matrix-org#14410](matrix-org#14410))
- Clean-up event persistence code. ([\matrix-org#14411](matrix-org#14411))
- Update docstring to clarify that `get_partial_state_events_batch` does not just give you completely arbitrary partial-state events. ([\matrix-org#14417](matrix-org#14417))
- Fix mypy errors introduced by bumping the locked version of `attrs` and `gitpython`. ([\matrix-org#14433](matrix-org#14433))
- Make Dependabot only bump Rust deps in the lock file. ([\matrix-org#14434](matrix-org#14434))
- Fix an incorrect stub return type for `PushRuleEvaluator.run`. ([\matrix-org#14451](matrix-org#14451))
- Improve performance of `/context` in large rooms. ([\matrix-org#14461](matrix-org#14461))

.ci/scripts/calculate_jobs.py

@@ -54,7 +54,7 @@ def set_output(key: str, value: str):
  {
  "python-version": "3.7",
  "database": "postgres",
- "postgres-version": "10",
+ "postgres-version": "11",
  "extras": "all",
  }
 ]
@@ -64,7 +64,7 @@ def set_output(key: str, value: str):
  {
  "python-version": "3.11",
  "database": "postgres",
- "postgres-version": "14",
+ "postgres-version": "15",
  "extras": "all",
  }
  )

.github/workflows/docs-pr-netlify.yaml

@@ -0,0 +1,34 @@
+name: Deploy documentation PR preview
+
+on:
+ workflow_run:
+ workflows: [ "Prepare documentation PR preview" ]
+ types:
+ - completed
+
+jobs:
+ netlify:
+ if: github.event.workflow_run.conclusion == 'success' && github.event.workflow_run.event == 'pull_request'
+ runs-on: ubuntu-latest
+ steps:
+ # There's a 'download artifact' action, but it hasn't been updated for the workflow_run action
+ # (https://github.com/actions/download-artifact/issues/60) so instead we get this mess:
+ - name: 📥 Download artifact
+ uses: dawidd6/action-download-artifact@e6e25ac3a2b93187502a8be1ef9e9603afc34925 # v2.24.2
+ with:
+ workflow: docs-pr.yaml
+ run_id: ${{ github.event.workflow_run.id }}
+ name: book
+ path: book
+
+ - name: 📤 Deploy to Netlify
+ uses: matrix-org/netlify-pr-preview@v1
+ with:
+ path: book
+ owner: ${{ github.event.workflow_run.head_repository.owner.login }}
+ branch: ${{ github.event.workflow_run.head_branch }}
+ revision: ${{ github.event.workflow_run.head_sha }}
+ token: ${{ secrets.NETLIFY_AUTH_TOKEN }}
+ site_id: ${{ secrets.NETLIFY_SITE_ID }}
+ desc: Documentation preview
+ deployment_env: PR Documentation Preview

.github/workflows/docs-pr.yaml

@@ -0,0 +1,34 @@
+name: Prepare documentation PR preview
+
+on:
+ pull_request:
+ paths:
+ - docs/**
+
+jobs:
+ pages:
+ name: GitHub Pages
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v2
+
+ - name: Setup mdbook
+ uses: peaceiris/actions-mdbook@adeb05db28a0c0004681db83893d56c0388ea9ea # v1.2.0
+ with:
+ mdbook-version: '0.4.17'
+
+ - name: Build the documentation
+ # mdbook will only create an index.html if we're including docs/README.md in SUMMARY.md.
+ # However, we're using docs/README.md for other purposes and need to pick a new page
+ # as the default. Let's opt for the welcome page instead.
+ run: |
+ mdbook build
+ cp book/welcome_and_overview.html book/index.html
+
+ - name: Upload Artifact
+ uses: actions/upload-artifact@v3
+ with:
+ name: book
+ path: book
+ # We'll only use this in a workflow_run, then we're done with it
+ retention-days: 1

.github/workflows/latest_deps.yml

@@ -27,10 +27,9 @@ jobs:
  steps:
  - uses: actions/checkout@v3
  - name: Install Rust
- uses: actions-rs/toolchain@v1
+ uses: dtolnay/rust-toolchain@55c7845fad90d0ae8b2e83715cb900e5e861e8cb
  with:
- toolchain: stable
- override: true
+ toolchain: stable
  - uses: Swatinem/rust-cache@v2
 
  # The dev dependencies aren't exposed in the wheel metadata (at least with current
@@ -62,10 +61,9 @@ jobs:
  - uses: actions/checkout@v3
 
  - name: Install Rust
- uses: actions-rs/toolchain@v1
+ uses: dtolnay/rust-toolchain@55c7845fad90d0ae8b2e83715cb900e5e861e8cb
  with:
- toolchain: stable
- override: true
+ toolchain: stable
  - uses: Swatinem/rust-cache@v2
 
  - run: sudo apt-get -qq install xmlsec1
@@ -136,10 +134,9 @@ jobs:
  - uses: actions/checkout@v3
 
  - name: Install Rust
- uses: actions-rs/toolchain@v1
+ uses: dtolnay/rust-toolchain@55c7845fad90d0ae8b2e83715cb900e5e861e8cb
  with:
- toolchain: stable
- override: true
+ toolchain: stable
  - uses: Swatinem/rust-cache@v2
 
  - name: Ensure sytest runs `pip install`

.github/workflows/tests.yml

@@ -33,6 +33,8 @@ jobs:
  steps:
  - uses: actions/checkout@v3
  - uses: actions/setup-python@v4
+ with:
+ python-version: "3.x"
  - uses: matrix-org/setup-python-poetry@v1
  with:
  extras: "all"
@@ -44,6 +46,8 @@ jobs:
  steps:
  - uses: actions/checkout@v3
  - uses: actions/setup-python@v4
+ with:
+ python-version: "3.x"
  - run: "pip install 'click==8.1.1' 'GitPython>=3.1.20'"
  - run: scripts-dev/check_schema_delta.py --force-colors
 
@@ -68,6 +72,8 @@ jobs:
  ref: ${{ github.event.pull_request.head.sha }}
  fetch-depth: 0
  - uses: actions/setup-python@v4
+ with:
+ python-version: "3.x"
  - run: "pip install 'towncrier>=18.6.0rc1'"
  - run: scripts-dev/check-newsfragment.sh
  env:
@@ -93,10 +99,12 @@ jobs:
  - uses: actions/checkout@v3
 
  - name: Install Rust
- uses: actions-rs/toolchain@v1
+ # There don't seem to be versioned releases of this action per se: for each rust
+ # version there is a branch which gets constantly rebased on top of master.
+ # We pin to a specific commit for paranoia's sake.
+ uses: dtolnay/rust-toolchain@55c7845fad90d0ae8b2e83715cb900e5e861e8cb
  with:
  toolchain: 1.58.1
- override: true
  components: clippy
  - uses: Swatinem/rust-cache@v2
 
@@ -111,11 +119,13 @@ jobs:
  - uses: actions/checkout@v3
 
  - name: Install Rust
- uses: actions-rs/toolchain@v1
+ # There don't seem to be versioned releases of this action per se: for each rust
+ # version there is a branch which gets constantly rebased on top of master.
+ # We pin to a specific commit for paranoia's sake.
+ uses: dtolnay/rust-toolchain@55c7845fad90d0ae8b2e83715cb900e5e861e8cb
  with:
- toolchain: 1.58.1
- override: true
- components: rustfmt
+ toolchain: 1.58.1
+ components: rustfmt
  - uses: Swatinem/rust-cache@v2
 
  - run: cargo fmt --check
@@ -143,6 +153,8 @@ jobs:
  steps:
  - uses: actions/checkout@v3
  - uses: actions/setup-python@v4
+ with:
+ python-version: "3.x"
  - id: get-matrix
  run: .ci/scripts/calculate_jobs.py
  outputs:
@@ -169,10 +181,12 @@ jobs:
  postgres:${{ matrix.job.postgres-version }}
 
  - name: Install Rust
- uses: actions-rs/toolchain@v1
+ # There don't seem to be versioned releases of this action per se: for each rust
+ # version there is a branch which gets constantly rebased on top of master.
+ # We pin to a specific commit for paranoia's sake.
+ uses: dtolnay/rust-toolchain@55c7845fad90d0ae8b2e83715cb900e5e861e8cb
  with:
  toolchain: 1.58.1
- override: true
  - uses: Swatinem/rust-cache@v2
 
  - uses: matrix-org/setup-python-poetry@v1
@@ -211,10 +225,12 @@ jobs:
  - uses: actions/checkout@v3
 
  - name: Install Rust
- uses: actions-rs/toolchain@v1
+ # There don't seem to be versioned releases of this action per se: for each rust
+ # version there is a branch which gets constantly rebased on top of master.
+ # We pin to a specific commit for paranoia's sake.
+ uses: dtolnay/rust-toolchain@55c7845fad90d0ae8b2e83715cb900e5e861e8cb
  with:
  toolchain: 1.58.1
- override: true
  - uses: Swatinem/rust-cache@v2
 
  # There aren't wheels for some of the older deps, so we need to install
@@ -327,10 +343,12 @@ jobs:
  run: cat sytest-blacklist .ci/worker-blacklist > synapse-blacklist-with-workers
 
  - name: Install Rust
- uses: actions-rs/toolchain@v1
+ # There don't seem to be versioned releases of this action per se: for each rust
+ # version there is a branch which gets constantly rebased on top of master.
+ # We pin to a specific commit for paranoia's sake.
+ uses: dtolnay/rust-toolchain@55c7845fad90d0ae8b2e83715cb900e5e861e8cb
  with:
  toolchain: 1.58.1
- override: true
  - uses: Swatinem/rust-cache@v2
 
  - name: Run SyTest
@@ -391,10 +409,10 @@ jobs:
  matrix:
  include:
  - python-version: "3.7"
- postgres-version: "10"
+ postgres-version: "11"
 
  - python-version: "3.11"
- postgres-version: "14"
+ postgres-version: "15"
 
  services:
  postgres:
@@ -412,6 +430,15 @@ jobs:
 
  steps:
  - uses: actions/checkout@v3
+ - name: Add PostgreSQL apt repository
+ # We need a version of pg_dump that can handle the version of
+ # PostgreSQL being tested against. The Ubuntu package repository lags
+ # behind new releases, so we have to use the PostreSQL apt repository.
+ # Steps taken from https://www.postgresql.org/download/linux/ubuntu/
+ run: |
+ sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
+ wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
+ sudo apt-get update
  - run: sudo apt-get -qq install xmlsec1 postgresql-client
  - uses: matrix-org/setup-python-poetry@v1
  with:
@@ -459,10 +486,12 @@ jobs:
  path: synapse
 
  - name: Install Rust
- uses: actions-rs/toolchain@v1
+ # There don't seem to be versioned releases of this action per se: for each rust
+ # version there is a branch which gets constantly rebased on top of master.
+ # We pin to a specific commit for paranoia's sake.
+ uses: dtolnay/rust-toolchain@55c7845fad90d0ae8b2e83715cb900e5e861e8cb
  with:
  toolchain: 1.58.1
- override: true
  - uses: Swatinem/rust-cache@v2
 
  - name: Prepare Complement's Prerequisites
@@ -485,10 +514,12 @@ jobs:
  - uses: actions/checkout@v3
 
  - name: Install Rust
- uses: actions-rs/toolchain@v1
+ # There don't seem to be versioned releases of this action per se: for each rust
+ # version there is a branch which gets constantly rebased on top of master.
+ # We pin to a specific commit for paranoia's sake.
+ uses: dtolnay/rust-toolchain@55c7845fad90d0ae8b2e83715cb900e5e861e8cb
  with:
  toolchain: 1.58.1
- override: true
  - uses: Swatinem/rust-cache@v2
 
  - run: cargo test

.github/workflows/triage_labelled.yml

@@ -11,34 +11,34 @@ jobs:
  if: >
  contains(github.event.issue.labels.*.name, 'X-Needs-Info')
  steps:
- - uses: octokit/graphql-action@v2.x
- id: add_to_project
+ - uses: actions/add-to-project@main
+ id: add_project
  with:
- headers: '{"GraphQL-Features": "projects_next_graphql"}'
- query: |
- mutation {
- updateProjectV2ItemFieldValue(
- input: {
- projectId: $projectid
- itemId: $contentid
- fieldId: $fieldid
- value: {
- singleSelectOptionId: "Todo"
+ project-url: "https://github.com/orgs/matrix-org/projects/67"
+ github-token: ${{ secrets.ELEMENT_BOT_TOKEN }}
+ - name: Set status
+ env:
+ GITHUB_TOKEN: ${{ secrets.ELEMENT_BOT_TOKEN }}
+ run: |
+ gh api graphql -f query='
+ mutation(
+ $project: ID!
+ $item: ID!
+ $fieldid: ID!
+ $columnid: String!
+ ) {
+ updateProjectV2ItemFieldValue(
+ input: {
+ projectId: $project
+ itemId: $item
+ fieldId: $fieldid
+ value: { 
+ singleSelectOptionId: $columnid
  }
- }
- ) {
- projectV2Item {
- id
- }
+ }
+ ) {
+ projectV2Item {
+ id
  }
  }
-
- projectid: ${{ env.PROJECT_ID }}
- contentid: ${{ github.event.issue.node_id }}
- fieldid: ${{ env.FIELD_ID }}
- optionid: ${{ env.OPTION_ID }}
- env:
- PROJECT_ID: "PVT_kwDOAIB0Bs4AFDdZ"
- GITHUB_TOKEN: ${{ secrets.ELEMENT_BOT_TOKEN }}
- FIELD_ID: "PVTSSF_lADOAIB0Bs4AFDdZzgC6ZA4"
- OPTION_ID: "ba22e43c"
+ }' -f project="PVT_kwDOAIB0Bs4AFDdZ" -f item=${{ steps.add_project.outputs.itemId }} -f fieldid="PVTSSF_lADOAIB0Bs4AFDdZzgC6ZA4" -f columnid=ba22e43c --silent