Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Commit

Permalink
Merge remote-tracking branch 'origin/develop' into shhs
Browse files Browse the repository at this point in the history
  • Loading branch information
hawkowl committed Jul 2, 2019
2 parents bed45ab + b491468 commit 4689408
Show file tree
Hide file tree
Showing 461 changed files with 17,745 additions and 11,322 deletions.
4 changes: 2 additions & 2 deletions .buildkite/pipeline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ steps:

- command:
- "python -m pip install tox"
- "tox -e pep8"
label: "\U0001F9F9 PEP-8"
- "tox -e check_codestyle"
label: "\U0001F9F9 Check Style"
plugins:
- docker#v3.0.1:
image: "python:3.6"
Expand Down
4 changes: 2 additions & 2 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ jobs:
machine: true
steps:
- checkout
- run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:${CIRCLE_TAG} -t matrixdotorg/synapse:${CIRCLE_TAG}-py3 --build-arg PYTHON_VERSION=3.6 .
- run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:${CIRCLE_TAG} -t matrixdotorg/synapse:${CIRCLE_TAG}-py3 .
- run: docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
- run: docker push matrixdotorg/synapse:${CIRCLE_TAG}
- run: docker push matrixdotorg/synapse:${CIRCLE_TAG}-py3
Expand All @@ -19,7 +19,7 @@ jobs:
machine: true
steps:
- checkout
- run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:latest -t matrixdotorg/synapse:latest-py3 --build-arg PYTHON_VERSION=3.6 .
- run: docker build -f docker/Dockerfile --label gitsha1=${CIRCLE_SHA1} -t matrixdotorg/synapse:latest -t matrixdotorg/synapse:latest-py3 .
- run: docker login --username $DOCKER_HUB_USERNAME --password $DOCKER_HUB_PASSWORD
- run: docker push matrixdotorg/synapse:latest
- run: docker push matrixdotorg/synapse:latest-py3
Expand Down
22 changes: 13 additions & 9 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -1,9 +1,13 @@
Dockerfile
.travis.yml
.gitignore
demo/etc
tox.ini
.git/*
.tox/*
debian/matrix-synapse/
debian/matrix-synapse-*/
# ignore everything by default
*

# things to include
!docker
!scripts
!synapse
!MANIFEST.in
!README.rst
!setup.py
!synctl

**/__pycache__
3 changes: 3 additions & 0 deletions AUTHORS.rst
Original file line number Diff line number Diff line change
Expand Up @@ -72,3 +72,6 @@ Jason Robinson <jasonr at matrix.org>

Joseph Weston <joseph at weston.cloud>
+ Add admin API for querying HS version

Benjamin Saunders <ben.e.saunders at gmail dot com>
* Documentation improvements
6 changes: 5 additions & 1 deletion CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,11 @@ Features
- Add a script to generate new signing-key files. ([\#5361](https://github.com/matrix-org/synapse/issues/5361))
- Update upgrade and installation guides ahead of 1.0. ([\#5371](https://github.com/matrix-org/synapse/issues/5371))
- Replace the `perspectives` configuration section with `trusted_key_servers`, and make validating the signatures on responses optional (since TLS will do this job for us). ([\#5374](https://github.com/matrix-org/synapse/issues/5374))
- Add ability to perform password reset via email without trusting the identity server. ([\#5377](https://github.com/matrix-org/synapse/issues/5377))
- Add ability to perform password reset via email without trusting the identity server. **As a result of this PR, password resets will now be disabled on the default configuration.**

Password reset emails are now sent from the homeserver by default, instead of the identity server. To enable this functionality, ensure `email` and `public_baseurl` config options are filled out.

If you would like to re-enable password resets being sent from the identity server (warning: this is dangerous! See [#5345](https://github.com/matrix-org/synapse/pull/5345)), set `email.trust_identity_server_for_password_resets` to true. ([\#5377](https://github.com/matrix-org/synapse/issues/5377))
- Set default room version to v4. ([\#5379](https://github.com/matrix-org/synapse/issues/5379))


Expand Down
31 changes: 18 additions & 13 deletions INSTALL.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
- [Choosing your server name](#choosing-your-server-name)
- [Installing Synapse](#installing-synapse)
- [Installing from source](#installing-from-source)
- [Platform-Specific Instructions](#platform-specific-instructions)
Expand All @@ -10,6 +11,22 @@
- [Setting up a TURN server](#setting-up-a-turn-server)
- [URL previews](#url-previews)

# Choosing your server name

It is important to choose the name for your server before you install Synapse,
because it cannot be changed later.

The server name determines the "domain" part of user-ids for users on your
server: these will all be of the format `@user:my.domain.name`. It also
determines how other matrix servers will reach yours for federation.

For a test configuration, set this to the hostname of your server. For a more
production-ready setup, you will probably want to specify your domain
(`example.com`) rather than a matrix-specific hostname here (in the same way
that your email address is probably `user@example.com` rather than
`user@email.example.com`) - but doing so may require more advanced setup: see
[Setting up Federation](docs/federate.md).

# Installing Synapse

## Installing from source
Expand Down Expand Up @@ -64,16 +81,7 @@ python -m synapse.app.homeserver \
--report-stats=[yes|no]
```

... substituting an appropriate value for `--server-name`. The server name
determines the "domain" part of user-ids for users on your server: these will
all be of the format `@user:my.domain.name`. It also determines how other
matrix servers will reach yours for Federation. For a test configuration,
set this to the hostname of your server. For a more production-ready setup, you
will probably want to specify your domain (`example.com`) rather than a
matrix-specific hostname here (in the same way that your email address is
probably `user@example.com` rather than `user@email.example.com`) - but
doing so may require more advanced setup: see [Setting up Federation](docs/federate.md).
Beware that the server name cannot be changed later.
... substituting an appropriate value for `--server-name`.

This command will generate you a config file that you can then customise, but it will
also generate a set of keys for you. These keys will allow your Home Server to
Expand All @@ -86,9 +94,6 @@ different. See the
[spec](https://matrix.org/docs/spec/server_server/latest.html#retrieving-server-keys)
for more information on key management.)

You will need to give Synapse a TLS certficate before it will start - see [TLS
certificates](#tls-certificates).

To actually run your new homeserver, pick a working directory for Synapse to
run (e.g. `~/synapse`), and::

Expand Down
32 changes: 25 additions & 7 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -340,8 +340,11 @@ log lines and looking for any 'Processed request' lines which take more than
a few seconds to execute. Please let us know at #synapse:matrix.org if
you see this failure mode so we can help debug it, however.

Help!! Synapse eats all my RAM!
-------------------------------
Help!! Synapse is slow and eats all my RAM/CPU!
-----------------------------------------------

First, ensure you are running the latest version of Synapse, using Python 3
with a PostgreSQL database.

Synapse's architecture is quite RAM hungry currently - we deliberately
cache a lot of recent room data and metadata in RAM in order to speed up
Expand All @@ -352,14 +355,29 @@ variable. The default is 0.5, which can be decreased to reduce RAM usage
in memory constrained enviroments, or increased if performance starts to
degrade.

However, degraded performance due to a low cache factor, common on
machines with slow disks, often leads to explosions in memory use due
backlogged requests. In this case, reducing the cache factor will make
things worse. Instead, try increasing it drastically. 2.0 is a good
starting value.

Using `libjemalloc <http://jemalloc.net/>`_ can also yield a significant
improvement in overall amount, and especially in terms of giving back RAM
to the OS. To use it, the library must simply be put in the LD_PRELOAD
environment variable when launching Synapse. On Debian, this can be done
by installing the ``libjemalloc1`` package and adding this line to
``/etc/default/matrix-synapse``::
improvement in overall memory use, and especially in terms of giving back
RAM to the OS. To use it, the library must simply be put in the
LD_PRELOAD environment variable when launching Synapse. On Debian, this
can be done by installing the ``libjemalloc1`` package and adding this
line to ``/etc/default/matrix-synapse``::

LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1

This can make a significant difference on Python 2.7 - it's unclear how
much of an improvement it provides on Python 3.x.

If you're encountering high CPU use by the Synapse process itself, you
may be affected by a bug with presence tracking that leads to a
massive excess of outgoing federation requests (see `discussion
<https://github.com/matrix-org/synapse/issues/3971>`_). If metrics
indicate that your server is also issuing far more outgoing federation
requests than can be accounted for by your users' activity, this is a
likely cause. The misbehavior can be worked around by setting
``use_presence: false`` in the Synapse config file.
1 change: 1 addition & 0 deletions changelog.d/4276.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Improve README section on performance troubleshooting.
1 change: 1 addition & 0 deletions changelog.d/5015.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add logging to 3pid invite signature verification.
1 change: 1 addition & 0 deletions changelog.d/5042.bugfix
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fix bug processing incoming events over federation if call to `/get_missing_events` fails.
1 change: 1 addition & 0 deletions changelog.d/5051.bugfix
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Prevent >1 room upgrades happening simultaneously on the same room.
1 change: 1 addition & 0 deletions changelog.d/5092.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Added possibilty to disable local password authentication. Contributed by Daniel Hoffend.
1 change: 1 addition & 0 deletions changelog.d/5313.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Update example haproxy config to a more compatible setup.
1 change: 1 addition & 0 deletions changelog.d/5475.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Synapse can now handle RestServlets that return coroutines.
1 change: 1 addition & 0 deletions changelog.d/5476.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add new metrics for number of forward extremities being persisted and number of state groups involved in resolution.
1 change: 1 addition & 0 deletions changelog.d/5480.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add an EXPERIMENTAL config option to try and periodically clean up extremities by sending dummy events.
1 change: 1 addition & 0 deletions changelog.d/5482.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Synapse's codebase is now formatted by `black`.
1 change: 1 addition & 0 deletions changelog.d/5490.bugfix
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fix failure to start under docker with SAML support enabled.
1 change: 1 addition & 0 deletions changelog.d/5493.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Track deactivated accounts in the database.
1 change: 1 addition & 0 deletions changelog.d/5498.bugfix
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fix intermittent exceptions on Apple hardware. Also fix bug that caused database activity times to be under-reported in log lines.
1 change: 1 addition & 0 deletions changelog.d/5499.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Some cleanups and sanity-checking in the CPU and database metrics.
1 change: 1 addition & 0 deletions changelog.d/5500.bugfix
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fix logging error when a tampered event is detected.
1 change: 1 addition & 0 deletions changelog.d/5502.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Improve email notification logging.
1 change: 1 addition & 0 deletions changelog.d/5505.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add support for handling pagination APIs on client reader worker.
1 change: 1 addition & 0 deletions changelog.d/5507.bugfix
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fix bug where clients could tight loop calling `/sync` for a period.
1 change: 1 addition & 0 deletions changelog.d/5509.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fix "Unexpected entry in 'full_schemas'" log warning.
1 change: 1 addition & 0 deletions changelog.d/5510.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Improve logging when generating config files.
1 change: 1 addition & 0 deletions changelog.d/5511.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Refactor and clean up Config parser for maintainability.
1 change: 1 addition & 0 deletions changelog.d/5512.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Improve help and cmdline option names for --generate-config options.
1 change: 1 addition & 0 deletions changelog.d/5513.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add support for handling pagination APIs on client reader worker.
1 change: 1 addition & 0 deletions changelog.d/5514.bugfix
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fix bug with `jinja2` preventing Synapse from starting. Users who had this problem should now simply need to run `pip install matrix-synapse`.
1 change: 1 addition & 0 deletions changelog.d/5516.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Allow configuration of the path used for ACME account keys.
1 change: 1 addition & 0 deletions changelog.d/5521.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Allow configuration of the path used for ACME account keys.
1 change: 1 addition & 0 deletions changelog.d/5522.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Allow configuration of the path used for ACME account keys.
1 change: 1 addition & 0 deletions changelog.d/5523.bugfix
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fix a regression where homeservers on private IP addresses were incorrectly blacklisted.
1 change: 1 addition & 0 deletions changelog.d/5524.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add --data-dir and --open-private-ports options.
1 change: 1 addition & 0 deletions changelog.d/5525.removal
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Remove support for cpu_affinity setting.
1 change: 1 addition & 0 deletions changelog.d/5531.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add support for handling pagination APIs on client reader worker.
1 change: 1 addition & 0 deletions changelog.d/5534.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Split public rooms directory auth config in two settings, in order to manage client auth independently from the federation part of it. Obsoletes the "restrict_public_rooms_to_local_users" configuration setting. If "restrict_public_rooms_to_local_users" is set in the config, Synapse will act as if both new options are enabled, i.e. require authentication through the client API and deny federation requests.
1 change: 1 addition & 0 deletions changelog.d/5537.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add information about how to install and run `black` on the codebase to code_style.rst.
1 change: 1 addition & 0 deletions changelog.d/5543.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Make the config clearer in that email.template_dir is relative to the Synapse's root directory, not the `synapse/` folder within it.
1 change: 1 addition & 0 deletions changelog.d/5545.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Update v1.0.0 release changelog to include more information about changes to password resets.
1 change: 1 addition & 0 deletions changelog.d/5546.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Update docker image to use Python 3.7.
1 change: 1 addition & 0 deletions changelog.d/5547.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Increase default log level for docker image to INFO. It can still be changed by editing the generated log.config file.
1 change: 1 addition & 0 deletions changelog.d/5548.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Remove non-functioning check_event_hash.py dev script.
1 change: 1 addition & 0 deletions changelog.d/5550.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The minimum TLS version used for outgoing federation requests can now be set with `federation_client_minimum_tls_version`.
1 change: 1 addition & 0 deletions changelog.d/5550.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Synapse will now only allow TLS v1.2 connections when serving federation, if it terminates TLS. As Synapse's allowed ciphers were only able to be used in TLSv1.2 before, this does not change behaviour.
1 change: 1 addition & 0 deletions changelog.d/5555.bugfix
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fixed m.login.jwt using unregistred user_id and added pyjwt>=1.6.4 as jwt conditional dependencies. Contributed by Pau Rodriguez-Estivill.
1 change: 1 addition & 0 deletions changelog.d/5557.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Logging when running GC collection on generation 0 is now at the DEBUG level, not INFO.
1 change: 1 addition & 0 deletions changelog.d/5558.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Improve install docs on choosing server_name.
1 change: 1 addition & 0 deletions changelog.d/5559.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Optimise devices changed query to not pull unnecessary rows from the database, reducing database load.
1 change: 1 addition & 0 deletions changelog.d/5561.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Update Docker image to deprecate the use of environment variables for configuration, and make the use of a static configuration the default.
1 change: 1 addition & 0 deletions changelog.d/5562.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Update Docker image to deprecate the use of environment variables for configuration, and make the use of a static configuration the default.
1 change: 1 addition & 0 deletions changelog.d/5563.bugfix
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Docker: Use a sensible location for data files when generating a config file.
1 change: 1 addition & 0 deletions changelog.d/5564.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Reduce the amount of stuff we send in the docker context.
1 change: 1 addition & 0 deletions changelog.d/5565.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Docker: Send synapse logs to the docker logging system, by default.
1 change: 1 addition & 0 deletions changelog.d/5566.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Update Docker image to deprecate the use of environment variables for configuration, and make the use of a static configuration the default.
1 change: 1 addition & 0 deletions changelog.d/5567.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Update Docker image to deprecate the use of environment variables for configuration, and make the use of a static configuration the default.
1 change: 1 addition & 0 deletions changelog.d/5568.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Docker image: open the non-TLS port by default.
1 change: 1 addition & 0 deletions changelog.d/5570.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Point the reverse links in the Purge History contrib scripts at the intended location.
1 change: 1 addition & 0 deletions changelog.d/5576.bugfix
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fix a bug that would cause invited users to receive several emails for a single 3PID invite in case the inviter is rate limited.
1 change: 1 addition & 0 deletions changelog.d/5585.misc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Synapse can now handle RestServlets that return coroutines.
Loading

0 comments on commit 4689408

Please sign in to comment.