Merge branch 'develop' of github.com:matrix-org/synapse into anoa/rem…

…ote_public_rooms_list_errors * 'develop' of github.com:matrix-org/synapse: (214 commits) Clean up some LoggingContext stuff (#7120) Fix CAS redirect url (#6634) Newsfile Fix starting workers when federation sending not split out. Use the proper error code when a canonical alias that does not exist is used. (#7109) Update pre-built package name for FreeBSD (#7107). (#7107) Convert `*StreamRow` classes to inner classes (#7116) matrix.org was fine Update CHANGES.md changelog typos 1.12.0 changelog 1.12.0 Fix processing of `groups` stream, and use symbolic names for streams (#7117) Set Referrer-Policy to no-referrer for media (#7009) Clean-up some auth/login REST code (#7115) Improve database configuration docs (#6988) Remove concept of a non-limited stream. (#7011) Convert some of the media REST code to async/await (#7110) Fix a bug in the federation API which could cause occasional "Failed to get PDU" errors (#7089). more changelog ...
matrix-org · Mar 25, 2020 · fc706cc · fc706cc
2 parents 092aef9 + b1cfaf0
commit fc706cc
Show file tree

Hide file tree

Showing 269 changed files with 7,055 additions and 5,080 deletions.
diff --git a/.buildkite/scripts/test_old_deps.sh b/.buildkite/scripts/test_old_deps.sh
@@ -6,12 +6,7 @@
 set -ex
 
 apt-get update
-apt-get install -y python3.5 python3.5-dev python3-pip libxml2-dev libxslt-dev zlib1g-dev
-
-# workaround for https://github.com/jaraco/zipp/issues/40
-python3.5 -m pip install 'setuptools>=34.4.0'
-
-python3.5 -m pip install tox
+apt-get install -y python3.5 python3.5-dev python3-pip libxml2-dev libxslt-dev zlib1g-dev tox
 
 export LANG="C.UTF-8"
 

diff --git a/CHANGES.md b/CHANGES.md
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -60,7 +60,7 @@ python 3.6 and to install each tool:
 
 ```
 # Install the dependencies
-pip install -U black flake8 isort
+pip install -U black flake8 flake8-comprehensions isort
 
 # Run the linter script
 ./scripts-dev/lint.sh

diff --git a/INSTALL.md b/INSTALL.md
@@ -124,12 +124,21 @@ sudo pacman -S base-devel python python-pip \
 
 #### CentOS/Fedora
 
-Installing prerequisites on CentOS 7 or Fedora 25:
+Installing prerequisites on CentOS 8 or Fedora>26:
+
+```
+sudo dnf install libtiff-devel libjpeg-devel libzip-devel freetype-devel \
+                 libwebp-devel tk-devel redhat-rpm-config \
+                 python3-virtualenv libffi-devel openssl-devel
+sudo dnf groupinstall "Development Tools"
+```
+
+Installing prerequisites on CentOS 7 or Fedora<=25:
 
 ```
 sudo yum install libtiff-devel libjpeg-devel libzip-devel freetype-devel \
                  lcms2-devel libwebp-devel tcl-devel tk-devel redhat-rpm-config \
-                 python-virtualenv libffi-devel openssl-devel
+                 python3-virtualenv libffi-devel openssl-devel
 sudo yum groupinstall "Development Tools"
 ```
 
@@ -374,7 +383,7 @@ Synapse can be found in the void repositories as 'synapse':
 Synapse can be installed via FreeBSD Ports or Packages contributed by Brendan Molloy from:
 
  - Ports: `cd /usr/ports/net-im/py-matrix-synapse && make install clean`
- - Packages: `pkg install py27-matrix-synapse`
+ - Packages: `pkg install py37-matrix-synapse`
 
 
 ### NixOS
@@ -388,15 +397,17 @@ Once you have installed synapse as above, you will need to configure it.
 
 ## TLS certificates
 
-The default configuration exposes a single HTTP port: http://localhost:8008. It
-is suitable for local testing, but for any practical use, you will either need
-to enable a reverse proxy, or configure Synapse to expose an HTTPS port.
+The default configuration exposes a single HTTP port on the local
+interface: `http://localhost:8008`. It is suitable for local testing,
+but for any practical use, you will need Synapse's APIs to be served
+over HTTPS.
 
-For information on using a reverse proxy, see
+The recommended way to do so is to set up a reverse proxy on port
+`8448`. You can find documentation on doing so in
 [docs/reverse_proxy.md](docs/reverse_proxy.md).
 
-To configure Synapse to expose an HTTPS port, you will need to edit
-`homeserver.yaml`, as follows:
+Alternatively, you can configure Synapse to expose an HTTPS port. To do
+so, you will need to edit `homeserver.yaml`, as follows:
 
 * First, under the `listeners` section, uncomment the configuration for the
   TLS-enabled listener. (Remove the hash sign (`#`) at the start of
@@ -414,10 +425,13 @@ To configure Synapse to expose an HTTPS port, you will need to edit
   point these settings at an existing certificate and key, or you can
   enable Synapse's built-in ACME (Let's Encrypt) support. Instructions
   for having Synapse automatically provision and renew federation
-  certificates through ACME can be found at [ACME.md](docs/ACME.md). If you
-  are using your own certificate, be sure to use a `.pem` file that includes
-  the full certificate chain including any intermediate certificates (for
-  instance, if using certbot, use `fullchain.pem` as your certificate, not
+  certificates through ACME can be found at [ACME.md](docs/ACME.md).
+  Note that, as pointed out in that document, this feature will not
+  work with installs set up after November 2019. 
+
+  If you are using your own certificate, be sure to use a `.pem` file that
+  includes the full certificate chain including any intermediate certificates
+  (for instance, if using certbot, use `fullchain.pem` as your certificate, not
   `cert.pem`).
 
 For a more detailed guide to configuring your server for federation, see

diff --git a/changelog.d/6634.bugfix b/changelog.d/6634.bugfix
@@ -0,0 +1 @@
+Fix single-sign on with CAS systems: pass the same service URL when requesting the CAS ticket and when calling the `proxyValidate` URL. Contributed by @Naugrimm.
diff --git a/changelog.d/6769.feature b/changelog.d/6769.feature
diff --git a/changelog.d/6781.bugfix b/changelog.d/6781.bugfix
diff --git a/changelog.d/6821.misc b/changelog.d/6821.misc
diff --git a/changelog.d/6823.misc b/changelog.d/6823.misc
diff --git a/changelog.d/6825.bugfix b/changelog.d/6825.bugfix
diff --git a/changelog.d/6827.misc b/changelog.d/6827.misc
diff --git a/changelog.d/6833.misc b/changelog.d/6833.misc
diff --git a/changelog.d/6836.misc b/changelog.d/6836.misc
diff --git a/changelog.d/6837.misc b/changelog.d/6837.misc
diff --git a/changelog.d/6840.misc b/changelog.d/6840.misc
diff --git a/changelog.d/6844.bugfix b/changelog.d/6844.bugfix
diff --git a/changelog.d/6846.doc b/changelog.d/6846.doc
diff --git a/changelog.d/6847.misc b/changelog.d/6847.misc
diff --git a/changelog.d/6849.bugfix b/changelog.d/6849.bugfix
diff --git a/changelog.d/6854.misc b/changelog.d/6854.misc
diff --git a/changelog.d/6855.misc b/changelog.d/6855.misc
diff --git a/changelog.d/6856.misc b/changelog.d/6856.misc
diff --git a/changelog.d/6857.misc b/changelog.d/6857.misc
diff --git a/changelog.d/6858.misc b/changelog.d/6858.misc
diff --git a/changelog.d/6862.misc b/changelog.d/6862.misc
diff --git a/changelog.d/6864.misc b/changelog.d/6864.misc
diff --git a/changelog.d/6866.feature b/changelog.d/6866.feature
diff --git a/changelog.d/6869.misc b/changelog.d/6869.misc
diff --git a/changelog.d/6871.misc b/changelog.d/6871.misc
diff --git a/changelog.d/6873.feature b/changelog.d/6873.feature
diff --git a/changelog.d/6882.misc b/changelog.d/6882.misc
diff --git a/changelog.d/6883.misc b/changelog.d/6883.misc
diff --git a/changelog.d/6887.misc b/changelog.d/6887.misc
diff --git a/changelog.d/6988.doc b/changelog.d/6988.doc
@@ -0,0 +1 @@
+Improve the documentation for database configuration.
diff --git a/changelog.d/7009.feature b/changelog.d/7009.feature
@@ -0,0 +1 @@
+Set `Referrer-Policy` header to `no-referrer` on media downloads.
diff --git a/changelog.d/7010.misc b/changelog.d/7010.misc
@@ -0,0 +1 @@
+Change device list streams to have one row per ID.
diff --git a/changelog.d/7011.misc b/changelog.d/7011.misc
@@ -0,0 +1 @@
+Remove concept of a non-limited stream.
diff --git a/changelog.d/7089.bugfix b/changelog.d/7089.bugfix
@@ -0,0 +1 @@
+Fix a bug in the federation API which could cause occasional "Failed to get PDU" errors.
diff --git a/changelog.d/7107.doc b/changelog.d/7107.doc
@@ -0,0 +1 @@
+Update pre-built package name for FreeBSD.
diff --git a/changelog.d/7109.bugfix b/changelog.d/7109.bugfix
@@ -0,0 +1 @@
+Return the proper error (M_BAD_ALIAS) when a non-existant canonical alias is provided.
diff --git a/changelog.d/7110.misc b/changelog.d/7110.misc
@@ -0,0 +1 @@
+Convert some of synapse.rest.media to async/await.
diff --git a/changelog.d/7115.misc b/changelog.d/7115.misc
@@ -0,0 +1 @@
+De-duplicate / remove unused REST code for login and auth.
diff --git a/changelog.d/7116.misc b/changelog.d/7116.misc
@@ -0,0 +1 @@
+Convert `*StreamRow` classes to inner classes.
diff --git a/changelog.d/7117.bugfix b/changelog.d/7117.bugfix
@@ -0,0 +1 @@
+Fix a bug which meant that groups updates were not correctly replicated between workers.
diff --git a/changelog.d/7120.misc b/changelog.d/7120.misc
@@ -0,0 +1 @@
+Clean up some LoggingContext code.
diff --git a/changelog.d/7133.bugfix b/changelog.d/7133.bugfix
@@ -0,0 +1 @@
+Fix starting workers when federation sending not split out.
diff --git a/contrib/docker/docker-compose.yml b/contrib/docker/docker-compose.yml
@@ -15,10 +15,9 @@ services:
     restart: unless-stopped
     # See the readme for a full documentation of the environment settings
     environment:
-      - SYNAPSE_CONFIG_PATH=/etc/homeserver.yaml
+      - SYNAPSE_CONFIG_PATH=/data/homeserver.yaml
     volumes:
       # You may either store all the files in a local folder
-      - ./matrix-config/homeserver.yaml:/etc/homeserver.yaml
       - ./files:/data
       # .. or you may split this between different storage points
       # - ./files:/data
@@ -56,6 +55,9 @@ services:
     environment:
       - POSTGRES_USER=synapse
       - POSTGRES_PASSWORD=changeme
+      # ensure the database gets created correctly
+      # https://github.com/matrix-org/synapse/blob/master/docs/postgres.md#set-up-database
+      - POSTGRES_INITDB_ARGS=--encoding=UTF-8 --lc-collate=C --lc-ctype=C
     volumes:
       # You may store the database tables in a local folder..
       - ./schemas:/var/lib/postgresql/data

diff --git a/contrib/grafana/README.md b/contrib/grafana/README.md
@@ -1,6 +1,6 @@
 # Using the Synapse Grafana dashboard
 
 0. Set up Prometheus and Grafana. Out of scope for this readme. Useful documentation about using Grafana with Prometheus: http://docs.grafana.org/features/datasources/prometheus/
-1. Have your Prometheus scrape your Synapse. https://github.com/matrix-org/synapse/blob/master/docs/metrics-howto.rst
+1. Have your Prometheus scrape your Synapse. https://github.com/matrix-org/synapse/blob/master/docs/metrics-howto.md
 2. Import dashboard into Grafana. Download `synapse.json`. Import it to Grafana and select the correct Prometheus datasource. http://docs.grafana.org/reference/export_import/
 3. Set up additional recording rules