Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Add in preliminary PyPy support #9123

Merged
merged 16 commits into from
Feb 4, 2021
Merged

Conversation

ShadowJonathan
Copy link
Contributor

@ShadowJonathan ShadowJonathan commented Jan 14, 2021

Pull Request Checklist

  • Pull request is based on the develop branch
  • Pull request includes a changelog file. The entry should:
    • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
    • Use markdown where necessary, mostly for code blocks.
    • End with either a period (.) or an exclamation mark (!).
    • Start with a capital letter.
  • Pull request includes a sign off
  • Code style is correct (run the linters)

Related to #8888
Fixes #5054

Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>

@ShadowJonathan
Copy link
Contributor Author

Blocked by pallets/markupsafe#176 (Unicode C API messing up), because this affects jinja2.escape, which is pretty vital to jinja templating, and so I wouldn't continue on with this if i wasn't sure this affected other areas of the server.

After they release version 2 (with hopefully this fixed), and jinja2 is verified to work with v2, I think adding "markupsafe>=2 ; platform_python_implementation != 'PyPy'" to dependencies will fix that issue.

@mattip
Copy link

mattip commented Jan 30, 2021

Can I see the failing log for PyPy? I want to exactly replicate the environment to debug the PyPy c-api call failure.

@ShadowJonathan
Copy link
Contributor Author

@mattip it's local, but the command is pypy3 -m tox -re pypy3 -- tests.rest.client.v1.test_login.MultiSSOTestCase.test_multi_sso_redirect, after installing tox on pypy3(.6), to invoke that failing test.

@mattip
Copy link

mattip commented Jan 30, 2021

It does not reproduce passes locally for me. What version of PyPy are you using?

@ShadowJonathan
Copy link
Contributor Author

@mattip

jonathan@jon:~$ pypy3 --version
Python 3.6.9 (7.3.1+dfsg-4ubuntu1, Sep 17 2020, 15:27:03)
[PyPy 7.3.1 with GCC 10.2.0]

@mattip
Copy link

mattip commented Jan 30, 2021

Ahh. Yes, there are unicode c-api bugs in that version. I guess that is the one available via apt get? The current version (7.3.3) would be the one you want to use. I see you are using a docker on circleCI for CI? You might want to use the docker images that cibuildwheel provides, they have up-to-date versions of python including pypy. Or you could move to github actions, they also provide up-to-date version of pypy. If you want to do the minimum change, you can use the release tarballs from https://downloads.python.org/pypy/. They are portable and should work on any machine that supports manylinux2010.

pushd /tmp
wget https://downloads.python.org/pypy/pypy3.6-v7.3.3-linux64.tar.bz2
tar -xf pypy3.6-v7.3.3-linux64.tar.bz2
export PATH=/tmp/pypy3.6-v7.3.3-linux64.tar/bin;$PATH
ln -s /tmp/pypy3.6-v7.3.3-linux64.tar/bin/python /tmp/pypy3.6-v7.3.3-linux64.tar/bin/pypy3
python -m ensurepip
python -m install tox 

or if you prefer conda

conda create -n pypy36 pypy3.6
conda activate pypy36
conda install -y tox

@ShadowJonathan
Copy link
Contributor Author

Docker image pypy:3.6 is PyPy 7.3.3, I think we'll be alright there.

Thanks for the headsup about the unicode bugs in that pypy version. 👍

I see you are using a docker on circleCI for CI?

Docker on buildkite, with boxes provided by matrix.org themselves, see the pipeline for more details.

@ShadowJonathan
Copy link
Contributor Author

ShadowJonathan commented Feb 1, 2021

Some test results (-postgres using psycopg2):

-j 2
py36: Ran 1501 tests in 875.411s
pypy3: Ran 1501 tests in 4378.433s

-j 3
py36-postgres: Ran 1501 tests in 706.022s
pypy3-postgres: Ran 1501 tests in 2548.308s

😐

@ShadowJonathan
Copy link
Contributor Author

https://www.pypy.org/performance.html tells that (perfect unit) tests and short-ran scripts don't get time to be optimised by the JIT

So tests are severely affected by the JIT this way, i'm going to look for ways to front-load this problem, and instead "warm up" the JIT to make it possible for tests to run more smoothly.

@ShadowJonathan
Copy link
Contributor Author

Looking further into it, i get the distinct impression pypy might have a C-API performance issue, this gist shows the performance difference between python and pypy on psycopg2.

I found pg8000 which is written in pure python, this might be counter-intuitive to use, but it's possible that with a JIT-optimized pypy, this library can outperform psycopg2, as with the tests, i saw them being done at blazing speeds when i removed a stray gc.collect() call somewhere, so this might be worth looking into.

@mattip
Copy link

mattip commented Feb 1, 2021

pypy might have a C-API performance issue

Definitely, emulating the C-API is slow. There is work ongoing to improve this via a new c-api (HPy) that will layer this on top of the current c-api.

Writing simple python code (as little dynamic class creation as possible, keeping class heirarchies shallow) will be fast on PyPy, for interfacing with C we recommend CFFI and for C++ we recommend cppyy.

@ShadowJonathan
Copy link
Contributor Author

ShadowJonathan commented Feb 1, 2021

Looking at https://morepypy.blogspot.com/2018/09/inside-cpyext-why-emulating-cpython-c.html, i got the idea to actually go use pg8000 (a pure-python DB API v2-complicit postgres driver). I'm currently running the tests again (at -j 3) using this driver, so we'll see how good the performance is, the early results (test OKs breezing past my screen) are promising.

Edit: Oh, i didnt see your comment @mattip, thanks! :D

@ShadowJonathan
Copy link
Contributor Author

Just finished running the new pypy3-postgres suite with -j 3 and pg8000 as driver: Ran 1501 tests in 763.756s :D

@ShadowJonathan
Copy link
Contributor Author

ShadowJonathan commented Feb 2, 2021

To reiterate: This PR will only add in some simple support for PyPy. In it's current form, a running server would be abysmally slow, due to C-API interaction with it's internal sqlite3 package, and C-API interaction with psycopg2cffi.

pg8000 will be added in another PR, and additional DB-cleaning-up PRs after that (that make adding extra adapters/drivers/engines easier, especially with the prospect of PEP 249 existing, and will seek to clean up the codebase to start to use with engine.new_connection() as conn: context managers and such). I'll also think about making a "postgres" virtual adapter then, for reverse compatibility, all "psycopg2"-specified driver configurations will keep using psycopg2, and wont suddenly switch over to pg8000 then, with "postgres" as "engine identifier", it'll switch between psycopg2 and pg8000 depending if the current running python is pypy or python. I think fancy things such as replicated writers will have to wait in pypy, as i first want to vet pg8000 before that happens.

I'll add the -prof cProfileing tox environment in another PR. (#9293)

(I'll open this PR once all tests pass)

@ShadowJonathan ShadowJonathan marked this pull request as ready for review February 2, 2021 09:23
@anoadragon453 anoadragon453 requested a review from a team February 2, 2021 11:20
@callahad
Copy link
Contributor

callahad commented Feb 2, 2021

In it's current form, a running server would be abysmally slow, due to C-API interaction with it's internal sqlite3 package, and C-API interaction with psycopg2cffi.

I believe you're misunderstanding; psycopg2cffi, per the name, uses CFFI, as mattip recommends. This is in contrast to CPython's native C extension API documented at https://docs.python.org/3/extending/extending.html.

@ShadowJonathan
Copy link
Contributor Author

It's still slow, the above test results (pypy3-postgres: Ran 1501 tests in 2548.308s) are using psycopg2cffi, so even if cffi is supposed to give a performance boost (or mitigate pypy's C API translation), I can't find evidence of it.

Copy link
Member

@clokep clokep left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks pretty reasonable overall! To be clear -- with these changes tests will pass? Did you run sytests by any chance?

synapse/python_dependencies.py Show resolved Hide resolved
synapse/storage/engines/__init__.py Outdated Show resolved Hide resolved
synapse/storage/engines/sqlite.py Show resolved Hide resolved
tests/__init__.py Outdated Show resolved Hide resolved
@mattip
Copy link

mattip commented Feb 3, 2021

Thanks for doing this. It would be interesting to see how this plays out in a production environment where the JIT can actually make things faster.

When thinking about which version of PyPy to test (assuming you will not test both python3.6 and python3.7), I would suggest you go with 3.7 not 3.6. Going forward, people are starting to drop 3.6 since it is no longer supported, and when underlying dependencies are updated they may not be available for 3.6.

@ShadowJonathan
Copy link
Contributor Author

To be clear -- with these changes tests will pass? Did you run sytests by any chance?

I ran tox -e pypy3 tests, but i haven't yet ran sytest tests, i'll do that in a second.

ShadowJonathan and others added 3 commits February 3, 2021 17:28
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
changelog.d/9123.misc Outdated Show resolved Hide resolved
tests/__init__.py Outdated Show resolved Hide resolved
tests/unittest.py Outdated Show resolved Hide resolved
ShadowJonathan and others added 5 commits February 3, 2021 18:04
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
@clokep clokep merged commit 2814028 into matrix-org:develop Feb 4, 2021
clokep added a commit that referenced this pull request Feb 19, 2021
Synapse 1.28.0rc1 (2021-02-19)
==============================

Note that this release drops support for ARMv7 in the official Docker images, due to repeated problems building for ARMv7 (and the associated maintenance burden this entails).

This release also fixes the documentation included in v1.27.0 around the callback URI for SAML2 identity providers. If your server is configured to use single sign-on via a SAML2 IdP, you may need to make configuration changes. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes.

Removal warning
---------------

The v1 list accounts API is deprecated and will be removed in a future release.
This API was undocumented and misleading. It can be replaced by the
[v2 list accounts API](https://github.com/matrix-org/synapse/blob/release-v1.28.0/docs/admin_api/user_admin_api.rst#list-accounts),
which has been available since Synapse 1.7.0 (2019-12-13).

Please check if you're using any scripts which use the admin API and replace
`GET /_synapse/admin/v1/users/<user_id>` with `GET /_synapse/admin/v2/users`.

Features
--------

- New admin API to get the context of an event: `/_synapse/admin/rooms/{roomId}/context/{eventId}`. ([\#9150](#9150))
- Further improvements to the user experience of registration via single sign-on. ([\#9300](#9300), [\#9301](#9301))
- Add hook to spam checker modules that allow checking file uploads and remote downloads. ([\#9311](#9311))
- Add support for receiving OpenID Connect authentication responses via form `POST`s rather than `GET`s. ([\#9376](#9376))
- Add the shadow-banning status to the admin API for user info. ([\#9400](#9400))

Bugfixes
--------

- Fix long-standing bug where sending email notifications would fail for rooms that the server had since left. ([\#9257](#9257))
- Fix bug in Synapse 1.27.0rc1 which meant the "session expired" error page during SSO registration was badly formatted. ([\#9296](#9296))
- Assert a maximum length for some parameters for spec compliance. ([\#9321](#9321), [\#9393](#9393))
- Fix additional errors when previewing URLs: "AttributeError 'NoneType' object has no attribute 'xpath'" and "ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.". ([\#9333](#9333))
- Fix a bug causing Synapse to impose the wrong type constraints on fields when processing responses from appservices to `/_matrix/app/v1/thirdparty/user/{protocol}`. ([\#9361](#9361))
- Fix bug where Synapse would occasionally stop reconnecting to Redis after the connection was lost. ([\#9391](#9391))
- Fix a long-standing bug when upgrading a room: "TypeError: '>' not supported between instances of 'NoneType' and 'int'". ([\#9395](#9395))
- Reduce the amount of memory used when generating the URL preview of a file that is larger than the `max_spider_size`. ([\#9421](#9421))
- Fix a long-standing bug in the deduplication of old presence, resulting in no deduplication. ([\#9425](#9425))
- The `ui_auth.session_timeout` config option can now be specified in terms of number of seconds/minutes/etc/. Contributed by Rishabh Arya. ([\#9426](#9426))
- Fix a bug introduced in v1.27.0: "TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType." related to the user directory. ([\#9428](#9428))

Updates to the Docker image
---------------------------

- Drop support for ARMv7 in Docker images. ([\#9433](#9433))

Improved Documentation
----------------------

- Reorganize CHANGELOG.md. ([\#9281](#9281))
- Add note to `auto_join_rooms` config option explaining existing rooms must be publicly joinable. ([\#9291](#9291))
- Correct name of Synapse's service file in TURN howto. ([\#9308](#9308))
- Fix the braces in the `oidc_providers` section of the sample config. ([\#9317](#9317))
- Update installation instructions on Fedora. ([\#9322](#9322))
- Add HTTP/2 support to the nginx example configuration. Contributed by David Vo. ([\#9390](#9390))
- Update docs for using Gitea as OpenID provider. ([\#9404](#9404))
- Document that pusher instances are shardable. ([\#9407](#9407))
- Fix erroneous documentation from v1.27.0 about updating the SAML2 callback URL. ([\#9434](#9434))

Deprecations and Removals
-------------------------

- Deprecate old admin API `GET /_synapse/admin/v1/users/<user_id>`. ([\#9429](#9429))

Internal Changes
----------------

- Fix 'object name reserved for internal use' errors with recent versions of SQLite. ([\#9003](#9003))
- Add experimental support for running Synapse with PyPy. ([\#9123](#9123))
- Deny access to additional IP addresses by default. ([\#9240](#9240))
- Update the `Cursor` type hints to better match PEP 249. ([\#9299](#9299))
- Add debug logging for SRV lookups. Contributed by @Bubu. ([\#9305](#9305))
- Improve logging for OIDC login flow. ([\#9307](#9307))
- Share the code for handling required attributes between the CAS and SAML handlers. ([\#9326](#9326))
- Clean up the code to load the metadata for OpenID Connect identity providers. ([\#9362](#9362))
- Convert tests to use `HomeserverTestCase`. ([\#9377](#9377), [\#9396](#9396))
- Update the version of black used to 20.8b1. ([\#9381](#9381))
- Allow OIDC config to override discovered values. ([\#9384](#9384))
- Remove some dead code from the acceptance of room invites path. ([\#9394](#9394))
- Clean up an unused method in the presence handler code. ([\#9408](#9408))
Half-Shot pushed a commit that referenced this pull request Feb 20, 2021
* Adds proper dependencies.
* Minor fixes in database layer.
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Mar 6, 2021
Synapse 1.28.0 (2021-02-25)
===========================

Note that this release drops support for ARMv7 in the official Docker images, due to repeated problems building for ARMv7 (and the associated maintenance burden this entails).

This release also fixes the documentation included in v1.27.0 around the callback URI for SAML2 identity providers. If your server is configured to use single sign-on via a SAML2 IdP, you may need to make configuration changes. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes.


Internal Changes
----------------

- Revert change in v1.28.0rc1 to remove the deprecated SAML endpoint. ([\#9474](matrix-org/synapse#9474))


Synapse 1.28.0rc1 (2021-02-19)
==============================

Removal warning
---------------

The v1 list accounts API is deprecated and will be removed in a future release.
This API was undocumented and misleading. It can be replaced by the
[v2 list accounts API](https://github.com/matrix-org/synapse/blob/release-v1.28.0/docs/admin_api/user_admin_api.rst#list-accounts),
which has been available since Synapse 1.7.0 (2019-12-13).

Please check if you're using any scripts which use the admin API and replace
`GET /_synapse/admin/v1/users/<user_id>` with `GET /_synapse/admin/v2/users`.


Features
--------

- New admin API to get the context of an event: `/_synapse/admin/rooms/{roomId}/context/{eventId}`. ([\#9150](matrix-org/synapse#9150))
- Further improvements to the user experience of registration via single sign-on. ([\#9300](matrix-org/synapse#9300), [\#9301](matrix-org/synapse#9301))
- Add hook to spam checker modules that allow checking file uploads and remote downloads. ([\#9311](matrix-org/synapse#9311))
- Add support for receiving OpenID Connect authentication responses via form `POST`s rather than `GET`s. ([\#9376](matrix-org/synapse#9376))
- Add the shadow-banning status to the admin API for user info. ([\#9400](matrix-org/synapse#9400))


Bugfixes
--------

- Fix long-standing bug where sending email notifications would fail for rooms that the server had since left. ([\#9257](matrix-org/synapse#9257))
- Fix bug introduced in Synapse 1.27.0rc1 which meant the "session expired" error page during SSO registration was badly formatted. ([\#9296](matrix-org/synapse#9296))
- Assert a maximum length for some parameters for spec compliance. ([\#9321](matrix-org/synapse#9321), [\#9393](matrix-org/synapse#9393))
- Fix additional errors when previewing URLs: "AttributeError 'NoneType' object has no attribute 'xpath'" and "ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.". ([\#9333](matrix-org/synapse#9333))
- Fix a bug causing Synapse to impose the wrong type constraints on fields when processing responses from appservices to `/_matrix/app/v1/thirdparty/user/{protocol}`. ([\#9361](matrix-org/synapse#9361))
- Fix bug where Synapse would occasionally stop reconnecting to Redis after the connection was lost. ([\#9391](matrix-org/synapse#9391))
- Fix a long-standing bug when upgrading a room: "TypeError: '>' not supported between instances of 'NoneType' and 'int'". ([\#9395](matrix-org/synapse#9395))
- Reduce the amount of memory used when generating the URL preview of a file that is larger than the `max_spider_size`. ([\#9421](matrix-org/synapse#9421))
- Fix a long-standing bug in the deduplication of old presence, resulting in no deduplication. ([\#9425](matrix-org/synapse#9425))
- The `ui_auth.session_timeout` config option can now be specified in terms of number of seconds/minutes/etc/. Contributed by Rishabh Arya. ([\#9426](matrix-org/synapse#9426))
- Fix a bug introduced in v1.27.0: "TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType." related to the user directory. ([\#9428](matrix-org/synapse#9428))


Updates to the Docker image
---------------------------

- Drop support for ARMv7 in Docker images. ([\#9433](matrix-org/synapse#9433))


Improved Documentation
----------------------

- Reorganize CHANGELOG.md. ([\#9281](matrix-org/synapse#9281))
- Add note to `auto_join_rooms` config option explaining existing rooms must be publicly joinable. ([\#9291](matrix-org/synapse#9291))
- Correct name of Synapse's service file in TURN howto. ([\#9308](matrix-org/synapse#9308))
- Fix the braces in the `oidc_providers` section of the sample config. ([\#9317](matrix-org/synapse#9317))
- Update installation instructions on Fedora. ([\#9322](matrix-org/synapse#9322))
- Add HTTP/2 support to the nginx example configuration. Contributed by David Vo. ([\#9390](matrix-org/synapse#9390))
- Update docs for using Gitea as OpenID provider. ([\#9404](matrix-org/synapse#9404))
- Document that pusher instances are shardable. ([\#9407](matrix-org/synapse#9407))
- Fix erroneous documentation from v1.27.0 about updating the SAML2 callback URL. ([\#9434](matrix-org/synapse#9434))


Deprecations and Removals
-------------------------

- Deprecate old admin API `GET /_synapse/admin/v1/users/<user_id>`. ([\#9429](matrix-org/synapse#9429))


Internal Changes
----------------

- Fix 'object name reserved for internal use' errors with recent versions of SQLite. ([\#9003](matrix-org/synapse#9003))
- Add experimental support for running Synapse with PyPy. ([\#9123](matrix-org/synapse#9123))
- Deny access to additional IP addresses by default. ([\#9240](matrix-org/synapse#9240))
- Update the `Cursor` type hints to better match PEP 249. ([\#9299](matrix-org/synapse#9299))
- Add debug logging for SRV lookups. Contributed by @Bubu. ([\#9305](matrix-org/synapse#9305))
- Improve logging for OIDC login flow. ([\#9307](matrix-org/synapse#9307))
- Share the code for handling required attributes between the CAS and SAML handlers. ([\#9326](matrix-org/synapse#9326))
- Clean up the code to load the metadata for OpenID Connect identity providers. ([\#9362](matrix-org/synapse#9362))
- Convert tests to use `HomeserverTestCase`. ([\#9377](matrix-org/synapse#9377), [\#9396](matrix-org/synapse#9396))
- Update the version of black used to 20.8b1. ([\#9381](matrix-org/synapse#9381))
- Allow OIDC config to override discovered values. ([\#9384](matrix-org/synapse#9384))
- Remove some dead code from the acceptance of room invites path. ([\#9394](matrix-org/synapse#9394))
- Clean up an unused method in the presence handler code. ([\#9408](matrix-org/synapse#9408))


Synapse 1.27.0 (2021-02-16)
===========================

Note that this release includes a change in Synapse to use Redis as a cache ─ as well as a pub/sub mechanism ─ if Redis support is enabled for workers. No action is needed by server administrators, and we do not expect resource usage of the Redis instance to change dramatically.

This release also changes the callback URI for OpenID Connect (OIDC) and SAML2 identity providers. If your server is configured to use single sign-on via an OIDC/OAuth2 or SAML2 IdP, you may need to make configuration changes. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes.

This release also changes escaping of variables in the HTML templates for SSO or email notifications. If you have customised these templates, please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes.


Bugfixes
--------

- Fix building Docker images for armv7. ([\#9405](matrix-org/synapse#9405))


Synapse 1.27.0rc2 (2021-02-11)
==============================

Features
--------

- Further improvements to the user experience of registration via single sign-on. ([\#9297](matrix-org/synapse#9297))


Bugfixes
--------

- Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](matrix-org/synapse#9302))
- Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](matrix-org/synapse#9313))


Improved Documentation
----------------------

- Clarify the sample configuration for changes made to the template loading code. ([\#9310](matrix-org/synapse#9310))


Synapse 1.27.0rc1 (2021-02-02)
==============================

Features
--------

- Add an admin API for getting and deleting forward extremities for a room. ([\#9062](matrix-org/synapse#9062))
- Add an admin API for retrieving the current room state of a room. ([\#9168](matrix-org/synapse#9168))
- Add experimental support for allowing clients to pick an SSO Identity Provider ([MSC2858](matrix-org/matrix-spec-proposals#2858)). ([\#9183](matrix-org/synapse#9183), [\#9242](matrix-org/synapse#9242))
- Add an admin API endpoint for shadow-banning users. ([\#9209](matrix-org/synapse#9209))
- Add ratelimits to the 3PID `/requestToken` APIs. ([\#9238](matrix-org/synapse#9238))
- Add support to the OpenID Connect integration for adding the user's email address. ([\#9245](matrix-org/synapse#9245))
- Add ratelimits to invites in rooms and to specific users. ([\#9258](matrix-org/synapse#9258))
- Improve the user experience of setting up an account via single-sign on. ([\#9262](matrix-org/synapse#9262), [\#9272](matrix-org/synapse#9272), [\#9275](matrix-org/synapse#9275), [\#9276](matrix-org/synapse#9276), [\#9277](matrix-org/synapse#9277), [\#9286](matrix-org/synapse#9286), [\#9287](matrix-org/synapse#9287))
- Add phone home stats for encrypted messages. ([\#9283](matrix-org/synapse#9283))
- Update the redirect URI for OIDC authentication. ([\#9288](matrix-org/synapse#9288))


Bugfixes
--------

- Fix spurious errors in logs when deleting a non-existant pusher. ([\#9121](matrix-org/synapse#9121))
- Fix a long-standing bug where Synapse would return a 500 error when a thumbnail did not exist (and auto-generation of thumbnails was not enabled). ([\#9163](matrix-org/synapse#9163))
- Fix a long-standing bug where an internal server error was raised when attempting to preview an HTML document in an unknown character encoding. ([\#9164](matrix-org/synapse#9164))
- Fix a long-standing bug where invalid data could cause errors when calculating the presentable room name for push. ([\#9165](matrix-org/synapse#9165))
- Fix bug where we sometimes didn't detect that Redis connections had died, causing workers to not see new data. ([\#9218](matrix-org/synapse#9218))
- Fix a bug where `None` was passed to Synapse modules instead of an empty dictionary if an empty module `config` block was provided in the homeserver config. ([\#9229](matrix-org/synapse#9229))
- Fix a bug in the `make_room_admin` admin API where it failed if the admin with the greatest power level was not in the room. Contributed by Pankaj Yadav. ([\#9235](matrix-org/synapse#9235))
- Prevent password hashes from getting dropped if a client failed threepid validation during a User Interactive Auth stage. Removes a workaround for an ancient bug in Riot Web <v0.7.4. ([\#9265](matrix-org/synapse#9265))
- Fix single-sign-on when the endpoints are routed to synapse workers. ([\#9271](matrix-org/synapse#9271))


Improved Documentation
----------------------

- Add docs for using Gitea as OpenID provider. ([\#9134](matrix-org/synapse#9134))
- Add link to Matrix VoIP tester for turn-howto. ([\#9135](matrix-org/synapse#9135))
- Add notes on integrating with Facebook for SSO login. ([\#9244](matrix-org/synapse#9244))


Deprecations and Removals
-------------------------

- The `service_url` parameter in `cas_config` is deprecated in favor of `public_baseurl`. ([\#9199](matrix-org/synapse#9199))
- Add new endpoint `/_synapse/client/saml2` for SAML2 authentication callbacks, and deprecate the old endpoint `/_matrix/saml2`. ([\#9289](matrix-org/synapse#9289))


Internal Changes
----------------

- Add tests to `test_user.UsersListTestCase` for List Users Admin API. ([\#9045](matrix-org/synapse#9045))
- Various improvements to the federation client. ([\#9129](matrix-org/synapse#9129))
- Speed up chain cover calculation when persisting a batch of state events at once. ([\#9176](matrix-org/synapse#9176))
- Add a `long_description_type` to the package metadata. ([\#9180](matrix-org/synapse#9180))
- Speed up batch insertion when using PostgreSQL. ([\#9181](matrix-org/synapse#9181), [\#9188](matrix-org/synapse#9188))
- Emit an error at startup if different Identity Providers are configured with the same `idp_id`. ([\#9184](matrix-org/synapse#9184))
- Improve performance of concurrent use of `StreamIDGenerators`. ([\#9190](matrix-org/synapse#9190))
- Add some missing source directories to the automatic linting script. ([\#9191](matrix-org/synapse#9191))
- Precompute joined hosts and store in Redis. ([\#9198](matrix-org/synapse#9198), [\#9227](matrix-org/synapse#9227))
- Clean-up template loading code. ([\#9200](matrix-org/synapse#9200))
- Fix the Python 3.5 old dependencies build. ([\#9217](matrix-org/synapse#9217))
- Update `isort` to v5.7.0 to bypass a bug where it would disagree with `black` about formatting. ([\#9222](matrix-org/synapse#9222))
- Add type hints to handlers code. ([\#9223](matrix-org/synapse#9223), [\#9232](matrix-org/synapse#9232))
- Fix Debian package building on Ubuntu 16.04 LTS (Xenial). ([\#9254](matrix-org/synapse#9254))
- Minor performance improvement during TLS handshake. ([\#9255](matrix-org/synapse#9255))
- Refactor the generation of summary text for email notifications. ([\#9260](matrix-org/synapse#9260))
- Restore PyPy compatibility by not calling CPython-specific GC methods when under PyPy. ([\#9270](matrix-org/synapse#9270))
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Workers running under PyPy attempt to load psycopg2 (not psycopg2cffi) and crash
4 participants